2024-04-24T01:47:29.096 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-24T01:47:29.096 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_yingxin@teuthology --description crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/simple thrashosds-health workloads/snaps-few-objects} --archive /home/teuthworker/archive/yingxin-2024-04-24_01:32:34-crimson-rados-ci-yingxin-crimson-load-obc-new-1-distro-default-smithi/7671138 --name yingxin-2024-04-24_01:32:34-crimson-rados-ci-yingxin-crimson-load-obc-new-1-distro-default-smithi --verbose -- /home/teuthworker/archive/yingxin-2024-04-24_01:32:34-crimson-rados-ci-yingxin-crimson-load-obc-new-1-distro-default-smithi/7671138/orig.config.yaml 2024-04-24T01:47:29.137 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-24T01:47:29.233 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yingxin-2024-04-24_01:32:34-crimson-rados-ci-yingxin-crimson-load-obc-new-1-distro-default-smithi/7671138 branch: ci-yingxin-crimson-load-obc-new-1 description: crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/simple thrashosds-health workloads/snaps-few-objects} email: yingxin.cheng@intel.com first_in_suite: false job_id: '7671138' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yingxin-2024-04-24_01:32:34-crimson-rados-ci-yingxin-crimson-load-obc-new-1-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: ci-yingxin-crimson-load-obc-new-1 ceph: conf: global: enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc osd_pool_default_min_size: 2 osd_pool_default_size: 3 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 osd pool default crimson: true paxos service trim min: 10 osd: bluestore block size: 96636764160 bluestore compression algorithm: snappy bluestore compression mode: aggressive bluestore fsck on mount: true bluestore rocksdb cf: false crimson alien thread cpu cores: 6-7 crimson osd obc lru size: 10 debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 err to stderr: true log flush on exit: true log to file: false log to stderr: true mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd debug reject backfill probability: 0.3 osd delete sleep: 1 osd failsafe full ratio: 0.95 osd max backfills: 3 osd max markdown count: 1000 osd objectstore: bluestore osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd snap trim sleep: 2 osd.0: crimson seastar cpu cores: 0-2 osd.1: crimson seastar cpu cores: 3-5 osd.2: crimson seastar cpu cores: 0-2 osd.3: crimson seastar cpu cores: 3-5 flavor: crimson fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(POOL_APP_NOT_ENABLED\) - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - mons down - mon down - out of quorum sha1: 424d8c4389b684b0a4d72510e1b4d398d8e6c8f4 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: flavor: crimson sha1: 424d8c4389b684b0a4d72510e1b4d398d8e6c8f4 selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 thrashosds: min_in: 4 workunit: branch: ci-yingxin-crimson-load-obc-new-1 env: CRIMSON_COMPAT: '1' sha1: 424d8c4389b684b0a4d72510e1b4d398d8e6c8f4 owner: scheduled_yingxin@teuthology priority: 60 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - osd.0 - osd.1 - client.0 - node-exporter.a - - mgr.x - osd.2 - osd.3 - client.1 - prometheus.a - node-exporter.b sha1: 424d8c4389b684b0a4d72510e1b4d398d8e6c8f4 sleep_before_teardown: 0 suite: crimson-rados suite_branch: ci-yingxin-crimson-load-obc-new-1 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_424d8c4389b684b0a4d72510e1b4d398d8e6c8f4/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 424d8c4389b684b0a4d72510e1b4d398d8e6c8f4 targets: smithi079.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBExTLpkVK6cnU8uvK6oiGFbBNE58CQpG+zqhnOeuD48hfXiTHMTNkdoxAlH5naPPwvvTytBB/PBvw2/kSuQQPGA= smithi103.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJwP7wGqBu8yx2tmUKdALm5FUtcVc092/3lmeqecFmPUCzjl9i77SkaJVwmYpo5zeySSoUJukG4ObFttE6oivLk= tasks: - install: null - ceph: conf: osd: debug monc: 20 flavor: crimson - thrashosds: ceph_objectstore_tool: false chance_down: 0 chance_inject_pause_short: 0 chance_reset_purged_snaps_last: 0 chance_thrash_cluster_full: 0 chance_thrash_pg_upmap: 0 dump_ops_enable: false min_in: 3 noscrub_toggle_delay: 0 reweight_osd: 0 sighup_delay: 0 thrash_primary_affinity: false timeout: 2400 - rados: clients: - client.0 objects: 50 op_weights: copy_from: 0 delete: 50 read: 100 rollback: 0 snap_create: 50 snap_remove: 50 write: 100 ops: 4000 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-24_01:32:34 tube: smithi user: yingxin verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-24T01:47:29.233 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_424d8c4389b684b0a4d72510e1b4d398d8e6c8f4/qa; will attempt to use it 2024-04-24T01:47:29.234 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_424d8c4389b684b0a4d72510e1b4d398d8e6c8f4/qa/tasks 2024-04-24T01:47:29.234 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-24T01:47:29.236 INFO:teuthology.task.internal:Checking packages... 2024-04-24T01:47:29.260 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'crimson' and ceph hash '424d8c4389b684b0a4d72510e1b4d398d8e6c8f4' 2024-04-24T01:47:29.260 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-24T01:47:29.260 INFO:teuthology.packaging:ref: None 2024-04-24T01:47:29.260 INFO:teuthology.packaging:tag: None 2024-04-24T01:47:29.260 INFO:teuthology.packaging:branch: ci-yingxin-crimson-load-obc-new-1 2024-04-24T01:47:29.260 INFO:teuthology.packaging:sha1: 424d8c4389b684b0a4d72510e1b4d398d8e6c8f4 2024-04-24T01:47:29.260 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&ref=ci-yingxin-crimson-load-obc-new-1 2024-04-24T01:47:29.486 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3227.g424d8c43 2024-04-24T01:47:29.487 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-24T01:47:29.494 INFO:teuthology.task.internal:no buildpackages task found 2024-04-24T01:47:29.494 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-24T01:47:29.515 INFO:teuthology.task.internal:Saving configuration 2024-04-24T01:47:29.531 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-24T01:47:29.537 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-24T01:47:29.564 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi079.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yingxin-2024-04-24_01:32:34-crimson-rados-ci-yingxin-crimson-load-obc-new-1-distro-default-smithi/7671138', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-24 01:42:05.650772', 'locked_by': 'scheduled_yingxin@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBExTLpkVK6cnU8uvK6oiGFbBNE58CQpG+zqhnOeuD48hfXiTHMTNkdoxAlH5naPPwvvTytBB/PBvw2/kSuQQPGA='} 2024-04-24T01:47:29.586 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi103.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yingxin-2024-04-24_01:32:34-crimson-rados-ci-yingxin-crimson-load-obc-new-1-distro-default-smithi/7671138', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-24 01:42:05.652403', 'locked_by': 'scheduled_yingxin@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJwP7wGqBu8yx2tmUKdALm5FUtcVc092/3lmeqecFmPUCzjl9i77SkaJVwmYpo5zeySSoUJukG4ObFttE6oivLk='} 2024-04-24T01:47:29.586 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-24T01:47:29.593 INFO:teuthology.task.internal:roles: ubuntu@smithi079.front.sepia.ceph.com - ['mon.a', 'osd.0', 'osd.1', 'client.0', 'node-exporter.a'] 2024-04-24T01:47:29.593 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['mgr.x', 'osd.2', 'osd.3', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-04-24T01:47:29.593 INFO:teuthology.run_tasks:Running task console_log... 2024-04-24T01:47:29.669 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f52848b9e50>, signals=[15]) 2024-04-24T01:47:29.670 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-24T01:47:29.676 INFO:teuthology.task.internal:Opening connections... 2024-04-24T01:47:29.677 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi079.front.sepia.ceph.com 2024-04-24T01:47:29.679 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi079.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T01:47:29.758 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi103.front.sepia.ceph.com 2024-04-24T01:47:29.759 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T01:47:29.837 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-24T01:47:29.844 DEBUG:teuthology.orchestra.run.smithi079:> uname -m 2024-04-24T01:47:29.861 INFO:teuthology.orchestra.run.smithi079.stdout:x86_64 2024-04-24T01:47:29.862 DEBUG:teuthology.orchestra.run.smithi079:> cat /etc/os-release 2024-04-24T01:47:29.918 INFO:teuthology.orchestra.run.smithi079.stdout:NAME="CentOS Stream" 2024-04-24T01:47:29.918 INFO:teuthology.orchestra.run.smithi079.stdout:VERSION="9" 2024-04-24T01:47:29.918 INFO:teuthology.orchestra.run.smithi079.stdout:ID="centos" 2024-04-24T01:47:29.918 INFO:teuthology.orchestra.run.smithi079.stdout:ID_LIKE="rhel fedora" 2024-04-24T01:47:29.918 INFO:teuthology.orchestra.run.smithi079.stdout:VERSION_ID="9" 2024-04-24T01:47:29.918 INFO:teuthology.orchestra.run.smithi079.stdout:PLATFORM_ID="platform:el9" 2024-04-24T01:47:29.919 INFO:teuthology.orchestra.run.smithi079.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-24T01:47:29.919 INFO:teuthology.orchestra.run.smithi079.stdout:ANSI_COLOR="0;31" 2024-04-24T01:47:29.919 INFO:teuthology.orchestra.run.smithi079.stdout:LOGO="fedora-logo-icon" 2024-04-24T01:47:29.919 INFO:teuthology.orchestra.run.smithi079.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-24T01:47:29.919 INFO:teuthology.orchestra.run.smithi079.stdout:HOME_URL="https://centos.org/" 2024-04-24T01:47:29.919 INFO:teuthology.orchestra.run.smithi079.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-24T01:47:29.919 INFO:teuthology.orchestra.run.smithi079.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-24T01:47:29.919 INFO:teuthology.orchestra.run.smithi079.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-24T01:47:29.919 INFO:teuthology.lock.ops:Updating smithi079.front.sepia.ceph.com on lock server 2024-04-24T01:47:29.941 DEBUG:teuthology.orchestra.run.smithi103:> uname -m 2024-04-24T01:47:29.958 INFO:teuthology.orchestra.run.smithi103.stdout:x86_64 2024-04-24T01:47:29.959 DEBUG:teuthology.orchestra.run.smithi103:> cat /etc/os-release 2024-04-24T01:47:30.016 INFO:teuthology.orchestra.run.smithi103.stdout:NAME="CentOS Stream" 2024-04-24T01:47:30.017 INFO:teuthology.orchestra.run.smithi103.stdout:VERSION="9" 2024-04-24T01:47:30.017 INFO:teuthology.orchestra.run.smithi103.stdout:ID="centos" 2024-04-24T01:47:30.017 INFO:teuthology.orchestra.run.smithi103.stdout:ID_LIKE="rhel fedora" 2024-04-24T01:47:30.017 INFO:teuthology.orchestra.run.smithi103.stdout:VERSION_ID="9" 2024-04-24T01:47:30.017 INFO:teuthology.orchestra.run.smithi103.stdout:PLATFORM_ID="platform:el9" 2024-04-24T01:47:30.017 INFO:teuthology.orchestra.run.smithi103.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-24T01:47:30.017 INFO:teuthology.orchestra.run.smithi103.stdout:ANSI_COLOR="0;31" 2024-04-24T01:47:30.017 INFO:teuthology.orchestra.run.smithi103.stdout:LOGO="fedora-logo-icon" 2024-04-24T01:47:30.017 INFO:teuthology.orchestra.run.smithi103.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-24T01:47:30.017 INFO:teuthology.orchestra.run.smithi103.stdout:HOME_URL="https://centos.org/" 2024-04-24T01:47:30.017 INFO:teuthology.orchestra.run.smithi103.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-24T01:47:30.017 INFO:teuthology.orchestra.run.smithi103.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-24T01:47:30.017 INFO:teuthology.orchestra.run.smithi103.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-24T01:47:30.018 INFO:teuthology.lock.ops:Updating smithi103.front.sepia.ceph.com on lock server 2024-04-24T01:47:30.032 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-24T01:47:30.042 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-24T01:47:30.050 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-24T01:47:30.051 DEBUG:teuthology.orchestra.run.smithi079:> test '!' -e /home/ubuntu/cephtest 2024-04-24T01:47:30.053 DEBUG:teuthology.orchestra.run.smithi103:> test '!' -e /home/ubuntu/cephtest 2024-04-24T01:47:30.074 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-24T01:47:30.080 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-24T01:47:30.080 DEBUG:teuthology.orchestra.run.smithi079:> test -z $(ls -A /var/lib/ceph) 2024-04-24T01:47:30.109 DEBUG:teuthology.orchestra.run.smithi103:> test -z $(ls -A /var/lib/ceph) 2024-04-24T01:47:30.227 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-24T01:47:30.282 INFO:teuthology.run_tasks:Running task kernel... 2024-04-24T01:47:30.296 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-04-24T01:47:30.296 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mgr.x': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-04-24T01:47:30.296 DEBUG:teuthology.orchestra.run.smithi079:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-24T01:47:30.297 DEBUG:teuthology.orchestra.run.smithi103:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-24T01:47:30.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T01:47:30.313 DEBUG:teuthology.orchestra.run.smithi079:> uname -r 2024-04-24T01:47:30.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T01:47:30.314 DEBUG:teuthology.orchestra.run.smithi103:> uname -r 2024-04-24T01:47:30.370 INFO:teuthology.orchestra.run.smithi079.stdout:5.14.0-437.el9.x86_64 2024-04-24T01:47:30.370 INFO:teuthology.task.kernel:Running kernel on smithi079: 5.14.0-437.el9.x86_64 2024-04-24T01:47:30.370 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum install -y kernel 2024-04-24T01:47:30.372 INFO:teuthology.orchestra.run.smithi103.stdout:5.14.0-437.el9.x86_64 2024-04-24T01:47:30.372 INFO:teuthology.task.kernel:Running kernel on smithi103: 5.14.0-437.el9.x86_64 2024-04-24T01:47:30.372 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum install -y kernel 2024-04-24T01:47:32.673 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.0 MB 00:00 2024-04-24T01:47:35.141 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - AppStream 14 MB/s | 19 MB 00:01 2024-04-24T01:47:35.443 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS Stream 9 - BaseOS 2.4 MB/s | 8.0 MB 00:03 2024-04-24T01:47:37.702 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS Stream 9 - AppStream 16 MB/s | 19 MB 00:01 2024-04-24T01:47:41.516 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - CRB 2.2 MB/s | 6.2 MB 00:02 2024-04-24T01:47:42.243 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS Stream 9 - CRB 6.8 MB/s | 6.2 MB 00:00 2024-04-24T01:47:44.296 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS Stream 9 - Extras packages 18 kB/s | 16 kB 00:00 2024-04-24T01:47:44.862 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - Extras packages 8.2 kB/s | 16 kB 00:01 2024-04-24T01:47:46.905 INFO:teuthology.orchestra.run.smithi103.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-04-24T01:47:46.920 INFO:teuthology.orchestra.run.smithi079.stdout:Extra Packages for Enterprise Linux 8.3 MB/s | 21 MB 00:02 2024-04-24T01:47:51.977 INFO:teuthology.orchestra.run.smithi103.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-04-24T01:47:52.070 INFO:teuthology.orchestra.run.smithi079.stdout:lab-extras 11 kB/s | 1.7 kB 00:00 2024-04-24T01:47:53.850 INFO:teuthology.orchestra.run.smithi103.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-24T01:47:53.850 INFO:teuthology.orchestra.run.smithi103.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-04-24T01:47:53.930 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-04-24T01:47:53.931 INFO:teuthology.orchestra.run.smithi103.stdout:Nothing to do. 2024-04-24T01:47:53.931 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-04-24T01:47:53.951 INFO:teuthology.orchestra.run.smithi079.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-24T01:47:53.951 INFO:teuthology.orchestra.run.smithi079.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-04-24T01:47:54.043 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-04-24T01:47:54.044 INFO:teuthology.orchestra.run.smithi079.stdout:Nothing to do. 2024-04-24T01:47:54.044 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-04-24T01:47:54.049 DEBUG:teuthology.orchestra.run.smithi103:> echo no | sudo yum reinstall kernel || true 2024-04-24T01:47:54.144 DEBUG:teuthology.orchestra.run.smithi079:> echo no | sudo yum reinstall kernel || true 2024-04-24T01:47:54.550 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:00:03 ago on Wed 24 Apr 2024 01:47:51 AM UTC. 2024-04-24T01:47:54.638 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:00:03 ago on Wed 24 Apr 2024 01:47:51 AM UTC. 2024-04-24T01:47:54.676 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-04-24T01:47:54.684 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-24T01:47:54.684 INFO:teuthology.orchestra.run.smithi103.stdout: Package Architecture Version Repository Size 2024-04-24T01:47:54.685 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-24T01:47:54.685 INFO:teuthology.orchestra.run.smithi103.stdout:Reinstalling: 2024-04-24T01:47:54.685 INFO:teuthology.orchestra.run.smithi103.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-24T01:47:54.685 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:47:54.685 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-04-24T01:47:54.685 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-24T01:47:54.685 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:47:54.685 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 255 k 2024-04-24T01:47:54.685 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 0 2024-04-24T01:47:54.685 INFO:teuthology.orchestra.run.smithi103.stderr:Operation aborted. 2024-04-24T01:47:54.745 INFO:teuthology.orchestra.run.smithi103.stdout:Is this ok [y/N]: 2024-04-24T01:47:54.746 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum reinstall -y kernel || true 2024-04-24T01:47:54.756 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-04-24T01:47:54.764 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-04-24T01:47:54.764 INFO:teuthology.orchestra.run.smithi079.stdout: Package Architecture Version Repository Size 2024-04-24T01:47:54.764 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-04-24T01:47:54.764 INFO:teuthology.orchestra.run.smithi079.stdout:Reinstalling: 2024-04-24T01:47:54.764 INFO:teuthology.orchestra.run.smithi079.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-24T01:47:54.764 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:47:54.764 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-04-24T01:47:54.764 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-04-24T01:47:54.764 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:47:54.764 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 255 k 2024-04-24T01:47:54.764 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 0 2024-04-24T01:47:54.765 INFO:teuthology.orchestra.run.smithi079.stderr:Operation aborted. 2024-04-24T01:47:54.824 INFO:teuthology.orchestra.run.smithi079.stdout:Is this ok [y/N]: 2024-04-24T01:47:54.825 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum reinstall -y kernel || true 2024-04-24T01:47:55.236 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:00:04 ago on Wed 24 Apr 2024 01:47:51 AM UTC. 2024-04-24T01:47:55.314 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:00:04 ago on Wed 24 Apr 2024 01:47:51 AM UTC. 2024-04-24T01:47:55.351 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-04-24T01:47:55.351 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-24T01:47:55.351 INFO:teuthology.orchestra.run.smithi103.stdout: Package Architecture Version Repository Size 2024-04-24T01:47:55.352 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-24T01:47:55.352 INFO:teuthology.orchestra.run.smithi103.stdout:Reinstalling: 2024-04-24T01:47:55.352 INFO:teuthology.orchestra.run.smithi103.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-24T01:47:55.352 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:47:55.352 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-04-24T01:47:55.352 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-24T01:47:55.352 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:47:55.352 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 255 k 2024-04-24T01:47:55.352 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 0 2024-04-24T01:47:55.352 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-04-24T01:47:55.430 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-04-24T01:47:55.431 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-04-24T01:47:55.431 INFO:teuthology.orchestra.run.smithi079.stdout: Package Architecture Version Repository Size 2024-04-24T01:47:55.431 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-04-24T01:47:55.431 INFO:teuthology.orchestra.run.smithi079.stdout:Reinstalling: 2024-04-24T01:47:55.431 INFO:teuthology.orchestra.run.smithi079.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-24T01:47:55.431 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:47:55.431 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-04-24T01:47:55.431 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-04-24T01:47:55.431 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:47:55.431 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 255 k 2024-04-24T01:47:55.431 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 0 2024-04-24T01:47:55.431 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-04-24T01:47:55.638 INFO:teuthology.orchestra.run.smithi103.stdout:kernel-5.14.0-437.el9.x86_64.rpm 1.1 MB/s | 255 kB 00:00 2024-04-24T01:47:55.638 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-04-24T01:47:55.638 INFO:teuthology.orchestra.run.smithi103.stdout:Total 891 kB/s | 255 kB 00:00 2024-04-24T01:47:55.654 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-04-24T01:47:55.730 INFO:teuthology.orchestra.run.smithi079.stdout:kernel-5.14.0-437.el9.x86_64.rpm 1.1 MB/s | 255 kB 00:00 2024-04-24T01:47:55.730 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-04-24T01:47:55.730 INFO:teuthology.orchestra.run.smithi079.stdout:Total 853 kB/s | 255 kB 00:00 2024-04-24T01:47:55.752 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-04-24T01:47:55.840 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-04-24T01:47:55.840 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-04-24T01:47:55.845 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-04-24T01:47:55.845 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-04-24T01:47:55.935 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-04-24T01:47:55.936 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-04-24T01:47:55.941 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-04-24T01:47:55.941 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-04-24T01:47:55.989 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-04-24T01:47:56.064 INFO:teuthology.orchestra.run.smithi103.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-24T01:47:56.064 INFO:teuthology.orchestra.run.smithi103.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-24T01:47:56.080 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-04-24T01:47:56.156 INFO:teuthology.orchestra.run.smithi079.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-24T01:47:56.156 INFO:teuthology.orchestra.run.smithi079.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-24T01:47:56.721 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-04-24T01:47:56.722 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-24T01:47:56.824 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-04-24T01:47:56.824 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-24T01:47:57.459 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-24T01:47:57.459 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:47:57.459 INFO:teuthology.orchestra.run.smithi103.stdout:Reinstalled: 2024-04-24T01:47:57.459 INFO:teuthology.orchestra.run.smithi103.stdout: kernel-5.14.0-437.el9.x86_64 2024-04-24T01:47:57.459 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:47:57.460 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-04-24T01:47:57.511 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-24T01:47:57.511 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:47:57.511 INFO:teuthology.orchestra.run.smithi079.stdout:Reinstalled: 2024-04-24T01:47:57.511 INFO:teuthology.orchestra.run.smithi079.stdout: kernel-5.14.0-437.el9.x86_64 2024-04-24T01:47:57.511 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:47:57.511 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-04-24T01:47:57.634 DEBUG:teuthology.orchestra.run.smithi103:> rpm -q kernel | sort -rV | head -n 1 2024-04-24T01:47:57.681 INFO:teuthology.orchestra.run.smithi103.stdout:kernel-5.14.0-437.el9.x86_64 2024-04-24T01:47:57.681 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-04-24T01:47:57.681 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-24T01:47:57.681 INFO:teuthology.task.kernel:Enabling kdb on mgr.x... 2024-04-24T01:47:57.681 DEBUG:teuthology.orchestra.run.smithi103:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-24T01:47:57.702 DEBUG:teuthology.orchestra.run.smithi079:> rpm -q kernel | sort -rV | head -n 1 2024-04-24T01:47:57.716 INFO:teuthology.orchestra.run.smithi103.stdout:ttyS1 2024-04-24T01:47:57.741 DEBUG:teuthology.parallel:result is None 2024-04-24T01:47:57.749 INFO:teuthology.orchestra.run.smithi079.stdout:kernel-5.14.0-437.el9.x86_64 2024-04-24T01:47:57.749 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-04-24T01:47:57.749 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-24T01:47:57.750 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-04-24T01:47:57.750 DEBUG:teuthology.orchestra.run.smithi079:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-24T01:47:57.776 INFO:teuthology.orchestra.run.smithi079.stdout:ttyS1 2024-04-24T01:47:57.810 DEBUG:teuthology.parallel:result is None 2024-04-24T01:47:57.810 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-24T01:47:57.817 INFO:teuthology.task.internal:Creating test directory... 2024-04-24T01:47:57.817 DEBUG:teuthology.orchestra.run.smithi079:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-24T01:47:57.852 DEBUG:teuthology.orchestra.run.smithi103:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-24T01:47:57.867 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-24T01:47:57.874 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-24T01:47:57.895 INFO:teuthology.task.internal:Creating archive directory... 2024-04-24T01:47:57.896 DEBUG:teuthology.orchestra.run.smithi079:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-24T01:47:57.907 DEBUG:teuthology.orchestra.run.smithi103:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-24T01:47:57.947 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-24T01:47:57.961 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-24T01:47:57.961 DEBUG:teuthology.orchestra.run.smithi079:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-24T01:47:57.986 DEBUG:teuthology.orchestra.run.smithi103:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-24T01:47:58.014 INFO:teuthology.orchestra.run.smithi103.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-24T01:47:58.022 INFO:teuthology.orchestra.run.smithi079.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-24T01:47:58.023 INFO:teuthology.orchestra.run.smithi103.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-24T01:47:58.032 INFO:teuthology.orchestra.run.smithi079.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-24T01:47:58.033 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-24T01:47:58.040 INFO:teuthology.task.internal:Configuring sudo... 2024-04-24T01:47:58.040 DEBUG:teuthology.orchestra.run.smithi079:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-24T01:47:58.076 DEBUG:teuthology.orchestra.run.smithi103:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-24T01:47:58.100 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-24T01:47:58.108 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-24T01:47:58.109 DEBUG:teuthology.orchestra.run.smithi079:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-24T01:47:58.139 DEBUG:teuthology.orchestra.run.smithi103:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-24T01:47:58.155 DEBUG:teuthology.orchestra.run.smithi079:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-24T01:47:58.231 DEBUG:teuthology.orchestra.run.smithi079:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-24T01:47:58.285 DEBUG:teuthology.orchestra.run.smithi079:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-24T01:47:58.367 DEBUG:teuthology.orchestra.run.smithi079:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-24T01:47:58.406 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-04-24T01:47:58.406 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-24T01:47:58.469 DEBUG:teuthology.orchestra.run.smithi103:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-24T01:47:58.511 DEBUG:teuthology.orchestra.run.smithi103:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-24T01:47:58.566 DEBUG:teuthology.orchestra.run.smithi103:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-24T01:47:58.647 DEBUG:teuthology.orchestra.run.smithi103:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-24T01:47:58.686 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-24T01:47:58.686 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-24T01:47:58.750 DEBUG:teuthology.orchestra.run.smithi079:> sudo service rsyslog restart 2024-04-24T01:47:58.752 DEBUG:teuthology.orchestra.run.smithi103:> sudo service rsyslog restart 2024-04-24T01:47:58.783 INFO:teuthology.orchestra.run.smithi079.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-24T01:47:58.822 INFO:teuthology.orchestra.run.smithi103.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-24T01:47:59.137 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-24T01:47:59.145 INFO:teuthology.task.internal:Starting timer... 2024-04-24T01:47:59.145 INFO:teuthology.run_tasks:Running task pcp... 2024-04-24T01:47:59.157 INFO:teuthology.run_tasks:Running task selinux... 2024-04-24T01:47:59.166 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-04-24T01:47:59.166 DEBUG:teuthology.orchestra.run.smithi079:> sudo service auditd rotate 2024-04-24T01:47:59.219 INFO:teuthology.orchestra.run.smithi079.stdout:Rotating logs: 2024-04-24T01:47:59.221 DEBUG:teuthology.orchestra.run.smithi103:> sudo service auditd rotate 2024-04-24T01:47:59.277 INFO:teuthology.orchestra.run.smithi103.stdout:Rotating logs: 2024-04-24T01:47:59.279 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-24T01:47:59.279 DEBUG:teuthology.orchestra.run.smithi079:> /usr/sbin/getenforce 2024-04-24T01:47:59.327 INFO:teuthology.orchestra.run.smithi079.stdout:Permissive 2024-04-24T01:47:59.327 DEBUG:teuthology.orchestra.run.smithi103:> /usr/sbin/getenforce 2024-04-24T01:47:59.345 INFO:teuthology.orchestra.run.smithi103.stdout:Permissive 2024-04-24T01:47:59.345 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi079.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi103.front.sepia.ceph.com': 'permissive'} 2024-04-24T01:47:59.345 DEBUG:teuthology.orchestra.run.smithi079:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-24T01:47:59.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T01:47:59.369 DEBUG:teuthology.orchestra.run.smithi103:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-24T01:47:59.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T01:47:59.409 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-24T01:47:59.409 DEBUG:teuthology.orchestra.run.smithi079:> sudo /usr/sbin/setenforce permissive 2024-04-24T01:47:59.435 DEBUG:teuthology.orchestra.run.smithi103:> sudo /usr/sbin/setenforce permissive 2024-04-24T01:47:59.472 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-24T01:47:59.482 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-24T01:47:59.498 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-24T01:47:59.499 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi079.front.sepia.ceph.com,smithi103.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-24T01:51:26.554 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi079.front.sepia.ceph.com'), Remote(name='ubuntu@smithi103.front.sepia.ceph.com')] 2024-04-24T01:51:26.555 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi079.front.sepia.ceph.com' 2024-04-24T01:51:26.556 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi079.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T01:51:26.630 DEBUG:teuthology.orchestra.run.smithi079:> true 2024-04-24T01:51:26.709 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi079.front.sepia.ceph.com' 2024-04-24T01:51:26.710 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-04-24T01:51:26.710 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T01:51:26.780 DEBUG:teuthology.orchestra.run.smithi103:> true 2024-04-24T01:51:26.861 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-04-24T01:51:26.861 INFO:teuthology.run_tasks:Running task clock... 2024-04-24T01:51:26.871 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-24T01:51:26.871 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-24T01:51:26.872 DEBUG:teuthology.orchestra.run.smithi079:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-24T01:51:26.874 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-24T01:51:26.874 DEBUG:teuthology.orchestra.run.smithi103:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-24T01:51:26.900 INFO:teuthology.orchestra.run.smithi079.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-24T01:51:26.914 INFO:teuthology.orchestra.run.smithi079.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-24T01:51:26.927 INFO:teuthology.orchestra.run.smithi103.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-24T01:51:26.936 INFO:teuthology.orchestra.run.smithi079.stderr:sudo: ntpd: command not found 2024-04-24T01:51:26.940 INFO:teuthology.orchestra.run.smithi103.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-24T01:51:26.946 INFO:teuthology.orchestra.run.smithi079.stdout:506 Cannot talk to daemon 2024-04-24T01:51:26.959 INFO:teuthology.orchestra.run.smithi079.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-24T01:51:26.962 INFO:teuthology.orchestra.run.smithi103.stderr:sudo: ntpd: command not found 2024-04-24T01:51:26.972 INFO:teuthology.orchestra.run.smithi079.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-24T01:51:26.973 INFO:teuthology.orchestra.run.smithi103.stdout:506 Cannot talk to daemon 2024-04-24T01:51:26.987 INFO:teuthology.orchestra.run.smithi103.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-24T01:51:26.999 INFO:teuthology.orchestra.run.smithi103.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-24T01:51:27.030 INFO:teuthology.orchestra.run.smithi079.stderr:bash: line 1: ntpq: command not found 2024-04-24T01:51:27.035 INFO:teuthology.orchestra.run.smithi079.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-24T01:51:27.035 INFO:teuthology.orchestra.run.smithi079.stdout:=============================================================================== 2024-04-24T01:51:27.035 INFO:teuthology.orchestra.run.smithi079.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T01:51:27.035 INFO:teuthology.orchestra.run.smithi079.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T01:51:27.035 INFO:teuthology.orchestra.run.smithi079.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T01:51:27.035 INFO:teuthology.orchestra.run.smithi079.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T01:51:27.059 INFO:teuthology.orchestra.run.smithi103.stderr:bash: line 1: ntpq: command not found 2024-04-24T01:51:27.062 INFO:teuthology.orchestra.run.smithi103.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-24T01:51:27.063 INFO:teuthology.orchestra.run.smithi103.stdout:=============================================================================== 2024-04-24T01:51:27.063 INFO:teuthology.orchestra.run.smithi103.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T01:51:27.063 INFO:teuthology.orchestra.run.smithi103.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T01:51:27.063 INFO:teuthology.orchestra.run.smithi103.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T01:51:27.063 INFO:teuthology.orchestra.run.smithi103.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T01:51:27.064 INFO:teuthology.run_tasks:Running task install... 2024-04-24T01:51:27.072 DEBUG:teuthology.task.install:project ceph 2024-04-24T01:51:27.072 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'crimson', 'sha1': '424d8c4389b684b0a4d72510e1b4d398d8e6c8f4'}} 2024-04-24T01:51:27.072 DEBUG:teuthology.task.install:config {'flavor': 'crimson', 'sha1': '424d8c4389b684b0a4d72510e1b4d398d8e6c8f4'} 2024-04-24T01:51:27.072 INFO:teuthology.task.install:Using flavor: crimson 2024-04-24T01:51:27.078 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-24T01:51:27.078 INFO:teuthology.task.install:extra packages: [] 2024-04-24T01:51:27.078 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '424d8c4389b684b0a4d72510e1b4d398d8e6c8f4', 'tag': None, 'wait_for_package': False} 2024-04-24T01:51:27.079 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=424d8c4389b684b0a4d72510e1b4d398d8e6c8f4 2024-04-24T01:51:27.081 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '424d8c4389b684b0a4d72510e1b4d398d8e6c8f4', 'tag': None, 'wait_for_package': False} 2024-04-24T01:51:27.081 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=424d8c4389b684b0a4d72510e1b4d398d8e6c8f4 2024-04-24T01:51:27.222 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/ci-yingxin-crimson-load-obc-new-1/424d8c4389b684b0a4d72510e1b4d398d8e6c8f4/centos/9/flavors/crimson/ 2024-04-24T01:51:27.223 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3227.g424d8c43 2024-04-24T01:51:27.300 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/ci-yingxin-crimson-load-obc-new-1/424d8c4389b684b0a4d72510e1b4d398d8e6c8f4/centos/9/flavors/crimson/ 2024-04-24T01:51:27.300 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3227.g424d8c43 2024-04-24T01:51:27.389 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/ci-yingxin-crimson-load-obc-new-1/424d8c4389b684b0a4d72510e1b4d398d8e6c8f4/centos/9/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/ci-yingxin-crimson-load-obc-new-1/424d8c4389b684b0a4d72510e1b4d398d8e6c8f4/centos/9/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/ci-yingxin-crimson-load-obc-new-1/424d8c4389b684b0a4d72510e1b4d398d8e6c8f4/centos/9/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-24T01:51:27.389 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-04-24T01:51:27.389 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-24T01:51:27.407 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/ci-yingxin-crimson-load-obc-new-1/424d8c4389b684b0a4d72510e1b4d398d8e6c8f4/centos/9/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/ci-yingxin-crimson-load-obc-new-1/424d8c4389b684b0a4d72510e1b4d398d8e6c8f4/centos/9/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/ci-yingxin-crimson-load-obc-new-1/424d8c4389b684b0a4d72510e1b4d398d8e6c8f4/centos/9/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-24T01:51:27.407 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-24T01:51:27.407 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-24T01:51:27.413 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-04-24T01:51:27.413 DEBUG:teuthology.orchestra.run.smithi079:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/424d8c4389b684b0a4d72510e1b4d398d8e6c8f4/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-24T01:51:27.432 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-04-24T01:51:27.432 DEBUG:teuthology.orchestra.run.smithi103:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/424d8c4389b684b0a4d72510e1b4d398d8e6c8f4/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-24T01:51:27.477 DEBUG:teuthology.orchestra.run.smithi079:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-24T01:51:27.497 DEBUG:teuthology.orchestra.run.smithi103:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-24T01:51:27.550 DEBUG:teuthology.orchestra.run.smithi079:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-24T01:51:27.569 DEBUG:teuthology.orchestra.run.smithi103:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-24T01:51:27.613 INFO:teuthology.orchestra.run.smithi079.stdout:check_obsoletes = 1 2024-04-24T01:51:27.614 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum clean all 2024-04-24T01:51:27.632 INFO:teuthology.orchestra.run.smithi103.stdout:check_obsoletes = 1 2024-04-24T01:51:27.634 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum clean all 2024-04-24T01:51:27.836 INFO:teuthology.orchestra.run.smithi079.stdout:48 files removed 2024-04-24T01:51:27.859 INFO:teuthology.orchestra.run.smithi103.stdout:48 files removed 2024-04-24T01:51:27.862 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-radosgw 2024-04-24T01:51:27.887 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-radosgw 2024-04-24T01:51:28.469 INFO:teuthology.orchestra.run.smithi079.stdout:ceph packages for x86_64 250 kB/s | 87 kB 00:00 2024-04-24T01:51:28.536 INFO:teuthology.orchestra.run.smithi103.stdout:ceph packages for x86_64 224 kB/s | 87 kB 00:00 2024-04-24T01:51:28.895 INFO:teuthology.orchestra.run.smithi079.stdout:ceph noarch packages 31 kB/s | 12 kB 00:00 2024-04-24T01:51:28.961 INFO:teuthology.orchestra.run.smithi103.stdout:ceph noarch packages 31 kB/s | 12 kB 00:00 2024-04-24T01:51:29.262 INFO:teuthology.orchestra.run.smithi079.stdout:ceph source packages 6.1 kB/s | 2.0 kB 00:00 2024-04-24T01:51:29.320 INFO:teuthology.orchestra.run.smithi103.stdout:ceph source packages 6.3 kB/s | 2.0 kB 00:00 2024-04-24T01:51:29.704 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS Stream 9 - BaseOS 20 MB/s | 8.0 MB 00:00 2024-04-24T01:51:29.790 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - BaseOS 18 MB/s | 8.0 MB 00:00 2024-04-24T01:51:31.766 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - AppStream 21 MB/s | 19 MB 00:00 2024-04-24T01:51:37.025 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - CRB 3.8 MB/s | 6.2 MB 00:01 2024-04-24T01:51:39.620 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - Extras packages 11 kB/s | 16 kB 00:01 2024-04-24T01:51:41.597 INFO:teuthology.orchestra.run.smithi103.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-04-24T01:51:46.653 INFO:teuthology.orchestra.run.smithi103.stdout:lab-extras 25 kB/s | 1.7 kB 00:00 2024-04-24T01:51:48.768 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-04-24T01:51:48.770 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-24T01:51:48.770 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repository Size 2024-04-24T01:51:48.770 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-24T01:51:48.771 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-04-24T01:51:48.771 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-radosgw x86_64 2:19.0.0-3227.g424d8c43.el9 ceph 10 M 2024-04-24T01:51:48.771 INFO:teuthology.orchestra.run.smithi103.stdout:Upgrading: 2024-04-24T01:51:48.771 INFO:teuthology.orchestra.run.smithi103.stdout: librados2 x86_64 2:19.0.0-3227.g424d8c43.el9 ceph 3.4 M 2024-04-24T01:51:48.771 INFO:teuthology.orchestra.run.smithi103.stdout: librbd1 x86_64 2:19.0.0-3227.g424d8c43.el9 ceph 3.7 M 2024-04-24T01:51:48.771 INFO:teuthology.orchestra.run.smithi103.stdout:Installing dependencies: 2024-04-24T01:51:48.771 INFO:teuthology.orchestra.run.smithi103.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-24T01:51:48.771 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-base x86_64 2:19.0.0-3227.g424d8c43.el9 ceph 5.9 M 2024-04-24T01:51:48.771 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-common x86_64 2:19.0.0-3227.g424d8c43.el9 ceph 19 M 2024-04-24T01:51:48.771 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-selinux x86_64 2:19.0.0-3227.g424d8c43.el9 ceph 25 k 2024-04-24T01:51:48.771 INFO:teuthology.orchestra.run.smithi103.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-24T01:51:48.772 INFO:teuthology.orchestra.run.smithi103.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-24T01:51:48.772 INFO:teuthology.orchestra.run.smithi103.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-24T01:51:48.772 INFO:teuthology.orchestra.run.smithi103.stdout: libcephfs2 x86_64 2:19.0.0-3227.g424d8c43.el9 ceph 796 k 2024-04-24T01:51:48.772 INFO:teuthology.orchestra.run.smithi103.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-24T01:51:48.772 INFO:teuthology.orchestra.run.smithi103.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-24T01:51:48.772 INFO:teuthology.orchestra.run.smithi103.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-24T01:51:48.772 INFO:teuthology.orchestra.run.smithi103.stdout: libradosstriper1 x86_64 2:19.0.0-3227.g424d8c43.el9 ceph 548 k 2024-04-24T01:51:48.772 INFO:teuthology.orchestra.run.smithi103.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-24T01:51:48.772 INFO:teuthology.orchestra.run.smithi103.stdout: librgw2 x86_64 2:19.0.0-3227.g424d8c43.el9 ceph 5.2 M 2024-04-24T01:51:48.772 INFO:teuthology.orchestra.run.smithi103.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-24T01:51:48.772 INFO:teuthology.orchestra.run.smithi103.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-24T01:51:48.772 INFO:teuthology.orchestra.run.smithi103.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-24T01:51:48.772 INFO:teuthology.orchestra.run.smithi103.stdout: python3-ceph-argparse x86_64 2:19.0.0-3227.g424d8c43.el9 ceph 45 k 2024-04-24T01:51:48.773 INFO:teuthology.orchestra.run.smithi103.stdout: python3-ceph-common x86_64 2:19.0.0-3227.g424d8c43.el9 ceph 135 k 2024-04-24T01:51:48.773 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cephfs x86_64 2:19.0.0-3227.g424d8c43.el9 ceph 159 k 2024-04-24T01:51:48.773 INFO:teuthology.orchestra.run.smithi103.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-24T01:51:48.773 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rados x86_64 2:19.0.0-3227.g424d8c43.el9 ceph 315 k 2024-04-24T01:51:48.773 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rbd x86_64 2:19.0.0-3227.g424d8c43.el9 ceph 293 k 2024-04-24T01:51:48.773 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rgw x86_64 2:19.0.0-3227.g424d8c43.el9 ceph 99 k 2024-04-24T01:51:48.773 INFO:teuthology.orchestra.run.smithi103.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-24T01:51:48.773 INFO:teuthology.orchestra.run.smithi103.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-24T01:51:48.773 INFO:teuthology.orchestra.run.smithi103.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-24T01:51:48.773 INFO:teuthology.orchestra.run.smithi103.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-24T01:51:48.773 INFO:teuthology.orchestra.run.smithi103.stdout:Installing weak dependencies: 2024-04-24T01:51:48.773 INFO:teuthology.orchestra.run.smithi103.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-24T01:51:48.773 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:51:48.773 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-04-24T01:51:48.773 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-24T01:51:48.774 INFO:teuthology.orchestra.run.smithi103.stdout:Install 30 Packages 2024-04-24T01:51:48.774 INFO:teuthology.orchestra.run.smithi103.stdout:Upgrade 2 Packages 2024-04-24T01:51:48.774 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:51:48.774 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 59 M 2024-04-24T01:51:48.774 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-04-24T01:51:49.896 INFO:teuthology.orchestra.run.smithi103.stdout:(1/32): ceph-base-19.0.0-3227.g424d8c43.el9.x86 7.0 MB/s | 5.9 MB 00:00 2024-04-24T01:51:49.988 INFO:teuthology.orchestra.run.smithi103.stdout:(2/32): ceph-selinux-19.0.0-3227.g424d8c43.el9. 274 kB/s | 25 kB 00:00 2024-04-24T01:51:50.114 INFO:teuthology.orchestra.run.smithi103.stdout:(3/32): libcephfs2-19.0.0-3227.g424d8c43.el9.x8 6.2 MB/s | 796 kB 00:00 2024-04-24T01:51:50.221 INFO:teuthology.orchestra.run.smithi103.stdout:(4/32): libradosstriper1-19.0.0-3227.g424d8c43. 5.0 MB/s | 548 kB 00:00 2024-04-24T01:51:50.864 INFO:teuthology.orchestra.run.smithi103.stdout:(5/32): ceph-radosgw-19.0.0-3227.g424d8c43.el9. 5.6 MB/s | 10 MB 00:01 2024-04-24T01:51:50.947 INFO:teuthology.orchestra.run.smithi103.stdout:(6/32): librgw2-19.0.0-3227.g424d8c43.el9.x86_6 7.2 MB/s | 5.2 MB 00:00 2024-04-24T01:51:50.972 INFO:teuthology.orchestra.run.smithi103.stdout:(7/32): python3-ceph-argparse-19.0.0-3227.g424d 417 kB/s | 45 kB 00:00 2024-04-24T01:51:51.156 INFO:teuthology.orchestra.run.smithi103.stdout:(8/32): ceph-common-19.0.0-3227.g424d8c43.el9.x 9.2 MB/s | 19 MB 00:02 2024-04-24T01:51:51.181 INFO:teuthology.orchestra.run.smithi103.stdout:(9/32): python3-ceph-common-19.0.0-3227.g424d8c 577 kB/s | 135 kB 00:00 2024-04-24T01:51:51.206 INFO:teuthology.orchestra.run.smithi103.stdout:(10/32): python3-cephfs-19.0.0-3227.g424d8c43.e 681 kB/s | 159 kB 00:00 2024-04-24T01:51:51.231 INFO:teuthology.orchestra.run.smithi103.stdout:(11/32): python3-rados-19.0.0-3227.g424d8c43.el 4.1 MB/s | 315 kB 00:00 2024-04-24T01:51:51.257 INFO:teuthology.orchestra.run.smithi103.stdout:(12/32): python3-rbd-19.0.0-3227.g424d8c43.el9. 3.8 MB/s | 293 kB 00:00 2024-04-24T01:51:51.282 INFO:teuthology.orchestra.run.smithi103.stdout:(13/32): python3-rgw-19.0.0-3227.g424d8c43.el9. 1.3 MB/s | 99 kB 00:00 2024-04-24T01:51:51.358 INFO:teuthology.orchestra.run.smithi103.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 851 kB/s | 107 kB 00:00 2024-04-24T01:51:51.399 INFO:teuthology.orchestra.run.smithi103.stdout:(15/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.1 MB/s | 160 kB 00:00 2024-04-24T01:51:51.417 INFO:teuthology.orchestra.run.smithi103.stdout:(16/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 337 kB/s | 45 kB 00:00 2024-04-24T01:51:51.467 INFO:teuthology.orchestra.run.smithi103.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.3 MB/s | 292 kB 00:00 2024-04-24T01:51:51.492 INFO:teuthology.orchestra.run.smithi103.stdout:(18/32): librdkafka-1.6.1-102.el9.x86_64.rpm 4.8 MB/s | 662 kB 00:00 2024-04-24T01:51:51.502 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS Stream 9 - AppStream 939 kB/s | 19 MB 00:20 2024-04-24T01:51:51.517 INFO:teuthology.orchestra.run.smithi103.stdout:(19/32): python3-prettytable-0.7.2-27.el9.noarc 420 kB/s | 42 kB 00:00 2024-04-24T01:51:51.543 INFO:teuthology.orchestra.run.smithi103.stdout:(20/32): qatlib-24.02.0-1.el9.x86_64.rpm 2.9 MB/s | 221 kB 00:00 2024-04-24T01:51:51.568 INFO:teuthology.orchestra.run.smithi103.stdout:(21/32): qatlib-service-24.02.0-1.el9.x86_64.rp 717 kB/s | 36 kB 00:00 2024-04-24T01:51:51.593 INFO:teuthology.orchestra.run.smithi103.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 630 kB/s | 47 kB 00:00 2024-04-24T01:51:51.985 INFO:teuthology.orchestra.run.smithi103.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 66 kB/s | 26 kB 00:00 2024-04-24T01:51:52.127 INFO:teuthology.orchestra.run.smithi103.stdout:(24/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 529 kB/s | 309 kB 00:00 2024-04-24T01:51:52.153 INFO:teuthology.orchestra.run.smithi103.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 290 kB/s | 49 kB 00:00 2024-04-24T01:51:52.261 INFO:teuthology.orchestra.run.smithi103.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 502 kB/s | 67 kB 00:00 2024-04-24T01:51:52.362 INFO:teuthology.orchestra.run.smithi103.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 1.9 MB/s | 192 kB 00:00 2024-04-24T01:51:52.470 INFO:teuthology.orchestra.run.smithi103.stdout:(28/32): libarrow-9.0.0-10.el9.x86_64.rpm 5.0 MB/s | 4.4 MB 00:00 2024-04-24T01:51:52.504 INFO:teuthology.orchestra.run.smithi103.stdout:(29/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.3 MB/s | 838 kB 00:00 2024-04-24T01:51:52.679 INFO:teuthology.orchestra.run.smithi103.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 5.0 MB/s | 1.6 MB 00:00 2024-04-24T01:51:52.863 INFO:teuthology.orchestra.run.smithi103.stdout:(31/32): librados2-19.0.0-3227.g424d8c43.el9.x8 8.7 MB/s | 3.4 MB 00:00 2024-04-24T01:51:52.988 INFO:teuthology.orchestra.run.smithi103.stdout:(32/32): librbd1-19.0.0-3227.g424d8c43.el9.x86_ 7.6 MB/s | 3.7 MB 00:00 2024-04-24T01:51:52.993 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-04-24T01:51:52.993 INFO:teuthology.orchestra.run.smithi103.stdout:Total 14 MB/s | 59 MB 00:04 2024-04-24T01:51:53.095 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-04-24T01:51:53.131 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-04-24T01:51:53.131 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-04-24T01:51:53.589 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-04-24T01:51:53.590 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-04-24T01:51:55.067 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-04-24T01:51:55.589 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-04-24T01:51:55.640 INFO:teuthology.orchestra.run.smithi103.stdout: Upgrading : librados2-2:19.0.0-3227.g424d8c43.el9.x86_64 2/34 2024-04-24T01:51:55.732 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: librados2-2:19.0.0-3227.g424d8c43.el9.x86_64 2/34 2024-04-24T01:51:55.821 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-rados-2:19.0.0-3227.g424d8c43.el9.x86_64 3/34 2024-04-24T01:51:55.859 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libcephfs2-2:19.0.0-3227.g424d8c43.el9.x86_64 4/34 2024-04-24T01:51:55.921 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: libcephfs2-2:19.0.0-3227.g424d8c43.el9.x86_64 4/34 2024-04-24T01:51:55.959 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : liboath-2.6.7-2.el9.x86_64 5/34 2024-04-24T01:51:55.988 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/34 2024-04-24T01:51:56.014 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/34 2024-04-24T01:51:56.036 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 8/34 2024-04-24T01:51:56.060 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 8/34 2024-04-24T01:51:56.211 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 9/34 2024-04-24T01:51:56.244 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 9/34 2024-04-24T01:51:56.477 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS Stream 9 - CRB 4.2 MB/s | 6.2 MB 00:01 2024-04-24T01:51:56.591 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 10/34 2024-04-24T01:51:56.649 INFO:teuthology.orchestra.run.smithi103.stdout: Upgrading : librbd1-2:19.0.0-3227.g424d8c43.el9.x86_64 11/34 2024-04-24T01:51:56.715 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: librbd1-2:19.0.0-3227.g424d8c43.el9.x86_64 11/34 2024-04-24T01:51:56.770 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-ceph-argparse-2:19.0.0-3227.g424d8c43.el9. 12/34 2024-04-24T01:51:56.812 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cephfs-2:19.0.0-3227.g424d8c43.el9.x86_64 13/34 2024-04-24T01:51:56.837 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-rbd-2:19.0.0-3227.g424d8c43.el9.x86_64 14/34 2024-04-24T01:51:56.914 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 15/34 2024-04-24T01:51:56.935 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libradosstriper1-2:19.0.0-3227.g424d8c43.el9.x86_6 16/34 2024-04-24T01:51:57.021 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3227.g424d8c43.el9.x86_6 16/34 2024-04-24T01:51:57.050 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : thrift-0.15.0-2.el9.x86_64 17/34 2024-04-24T01:51:57.157 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : re2-1:20211101-3.el9.x86_64 18/34 2024-04-24T01:51:57.193 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 19/34 2024-04-24T01:51:57.233 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 20/34 2024-04-24T01:51:57.258 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 21/34 2024-04-24T01:51:57.287 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 22/34 2024-04-24T01:51:57.318 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 23/34 2024-04-24T01:51:57.354 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-04-24T01:51:57.916 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-04-24T01:51:57.941 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : librgw2-2:19.0.0-3227.g424d8c43.el9.x86_64 26/34 2024-04-24T01:51:58.012 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: librgw2-2:19.0.0-3227.g424d8c43.el9.x86_64 26/34 2024-04-24T01:51:58.082 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-rgw-2:19.0.0-3227.g424d8c43.el9.x86_64 27/34 2024-04-24T01:51:58.110 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-ceph-common-2:19.0.0-3227.g424d8c43.el9.x8 28/34 2024-04-24T01:51:58.372 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS Stream 9 - Extras packages 22 kB/s | 16 kB 00:00 2024-04-24T01:52:00.554 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-common-2:19.0.0-3227.g424d8c43.el9.x86_64 29/34 2024-04-24T01:52:00.667 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-common-2:19.0.0-3227.g424d8c43.el9.x86_64 29/34 2024-04-24T01:52:01.328 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-common-2:19.0.0-3227.g424d8c43.el9.x86_64 29/34 2024-04-24T01:52:01.377 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-base-2:19.0.0-3227.g424d8c43.el9.x86_64 30/34 2024-04-24T01:52:01.469 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-base-2:19.0.0-3227.g424d8c43.el9.x86_64 30/34 2024-04-24T01:52:01.469 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-24T01:52:01.469 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-24T01:52:01.469 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:52:01.504 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-selinux-2:19.0.0-3227.g424d8c43.el9.x86_64 31/34 2024-04-24T01:52:02.945 INFO:teuthology.orchestra.run.smithi079.stdout:Extra Packages for Enterprise Linux 4.7 MB/s | 21 MB 00:04 2024-04-24T01:52:08.003 INFO:teuthology.orchestra.run.smithi079.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-04-24T01:52:10.120 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-04-24T01:52:10.123 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-04-24T01:52:10.123 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2024-04-24T01:52:10.123 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-04-24T01:52:10.123 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-04-24T01:52:10.123 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-radosgw x86_64 2:19.0.0-3227.g424d8c43.el9 ceph 10 M 2024-04-24T01:52:10.123 INFO:teuthology.orchestra.run.smithi079.stdout:Upgrading: 2024-04-24T01:52:10.123 INFO:teuthology.orchestra.run.smithi079.stdout: librados2 x86_64 2:19.0.0-3227.g424d8c43.el9 ceph 3.4 M 2024-04-24T01:52:10.123 INFO:teuthology.orchestra.run.smithi079.stdout: librbd1 x86_64 2:19.0.0-3227.g424d8c43.el9 ceph 3.7 M 2024-04-24T01:52:10.123 INFO:teuthology.orchestra.run.smithi079.stdout:Installing dependencies: 2024-04-24T01:52:10.124 INFO:teuthology.orchestra.run.smithi079.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-24T01:52:10.124 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-base x86_64 2:19.0.0-3227.g424d8c43.el9 ceph 5.9 M 2024-04-24T01:52:10.124 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-common x86_64 2:19.0.0-3227.g424d8c43.el9 ceph 19 M 2024-04-24T01:52:10.124 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-selinux x86_64 2:19.0.0-3227.g424d8c43.el9 ceph 25 k 2024-04-24T01:52:10.124 INFO:teuthology.orchestra.run.smithi079.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-24T01:52:10.124 INFO:teuthology.orchestra.run.smithi079.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-24T01:52:10.124 INFO:teuthology.orchestra.run.smithi079.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-24T01:52:10.124 INFO:teuthology.orchestra.run.smithi079.stdout: libcephfs2 x86_64 2:19.0.0-3227.g424d8c43.el9 ceph 796 k 2024-04-24T01:52:10.124 INFO:teuthology.orchestra.run.smithi079.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-24T01:52:10.124 INFO:teuthology.orchestra.run.smithi079.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-24T01:52:10.124 INFO:teuthology.orchestra.run.smithi079.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-24T01:52:10.125 INFO:teuthology.orchestra.run.smithi079.stdout: libradosstriper1 x86_64 2:19.0.0-3227.g424d8c43.el9 ceph 548 k 2024-04-24T01:52:10.125 INFO:teuthology.orchestra.run.smithi079.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-24T01:52:10.125 INFO:teuthology.orchestra.run.smithi079.stdout: librgw2 x86_64 2:19.0.0-3227.g424d8c43.el9 ceph 5.2 M 2024-04-24T01:52:10.125 INFO:teuthology.orchestra.run.smithi079.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-24T01:52:10.125 INFO:teuthology.orchestra.run.smithi079.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-24T01:52:10.125 INFO:teuthology.orchestra.run.smithi079.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-24T01:52:10.125 INFO:teuthology.orchestra.run.smithi079.stdout: python3-ceph-argparse x86_64 2:19.0.0-3227.g424d8c43.el9 ceph 45 k 2024-04-24T01:52:10.125 INFO:teuthology.orchestra.run.smithi079.stdout: python3-ceph-common x86_64 2:19.0.0-3227.g424d8c43.el9 ceph 135 k 2024-04-24T01:52:10.125 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cephfs x86_64 2:19.0.0-3227.g424d8c43.el9 ceph 159 k 2024-04-24T01:52:10.125 INFO:teuthology.orchestra.run.smithi079.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-24T01:52:10.125 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rados x86_64 2:19.0.0-3227.g424d8c43.el9 ceph 315 k 2024-04-24T01:52:10.125 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rbd x86_64 2:19.0.0-3227.g424d8c43.el9 ceph 293 k 2024-04-24T01:52:10.126 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rgw x86_64 2:19.0.0-3227.g424d8c43.el9 ceph 99 k 2024-04-24T01:52:10.126 INFO:teuthology.orchestra.run.smithi079.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-24T01:52:10.126 INFO:teuthology.orchestra.run.smithi079.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-24T01:52:10.126 INFO:teuthology.orchestra.run.smithi079.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-24T01:52:10.126 INFO:teuthology.orchestra.run.smithi079.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-24T01:52:10.126 INFO:teuthology.orchestra.run.smithi079.stdout:Installing weak dependencies: 2024-04-24T01:52:10.126 INFO:teuthology.orchestra.run.smithi079.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-24T01:52:10.126 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:52:10.126 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-04-24T01:52:10.126 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-04-24T01:52:10.126 INFO:teuthology.orchestra.run.smithi079.stdout:Install 30 Packages 2024-04-24T01:52:10.126 INFO:teuthology.orchestra.run.smithi079.stdout:Upgrade 2 Packages 2024-04-24T01:52:10.127 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:52:10.127 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 59 M 2024-04-24T01:52:10.127 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-04-24T01:52:11.308 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3227.g424d8c43.el9.x86_64 31/34 2024-04-24T01:52:11.309 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /sys 2024-04-24T01:52:11.309 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /proc 2024-04-24T01:52:11.309 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /mnt 2024-04-24T01:52:11.309 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /var/tmp 2024-04-24T01:52:11.309 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /home 2024-04-24T01:52:11.309 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /root 2024-04-24T01:52:11.309 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /tmp 2024-04-24T01:52:11.309 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:52:12.253 INFO:teuthology.orchestra.run.smithi079.stdout:(1/32): ceph-radosgw-19.0.0-3227.g424d8c43.el9. 6.8 MB/s | 10 MB 00:01 2024-04-24T01:52:12.295 INFO:teuthology.orchestra.run.smithi079.stdout:(2/32): ceph-selinux-19.0.0-3227.g424d8c43.el9. 606 kB/s | 25 kB 00:00 2024-04-24T01:52:12.406 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-radosgw-2:19.0.0-3227.g424d8c43.el9.x86_64 32/34 2024-04-24T01:52:12.430 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3227.g424d8c43.el9.x86_64 32/34 2024-04-24T01:52:12.430 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T01:52:12.430 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-24T01:52:12.430 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-24T01:52:12.430 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-24T01:52:12.430 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:52:12.432 INFO:teuthology.orchestra.run.smithi103.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-04-24T01:52:12.462 INFO:teuthology.orchestra.run.smithi079.stdout:(3/32): libcephfs2-19.0.0-3227.g424d8c43.el9.x8 4.7 MB/s | 796 kB 00:00 2024-04-24T01:52:12.517 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-04-24T01:52:12.517 INFO:teuthology.orchestra.run.smithi103.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-04-24T01:52:12.587 INFO:teuthology.orchestra.run.smithi079.stdout:(4/32): libradosstriper1-19.0.0-3227.g424d8c43. 4.3 MB/s | 548 kB 00:00 2024-04-24T01:52:12.796 INFO:teuthology.orchestra.run.smithi079.stdout:(5/32): ceph-common-19.0.0-3227.g424d8c43.el9.x 9.6 MB/s | 19 MB 00:02 2024-04-24T01:52:12.880 INFO:teuthology.orchestra.run.smithi079.stdout:(6/32): ceph-base-19.0.0-3227.g424d8c43.el9.x86 2.8 MB/s | 5.9 MB 00:02 2024-04-24T01:52:12.896 INFO:teuthology.orchestra.run.smithi079.stdout:(7/32): python3-ceph-argparse-19.0.0-3227.g424d 450 kB/s | 45 kB 00:00 2024-04-24T01:52:12.938 INFO:teuthology.orchestra.run.smithi079.stdout:(8/32): python3-ceph-common-19.0.0-3227.g424d8c 2.3 MB/s | 135 kB 00:00 2024-04-24T01:52:12.964 INFO:teuthology.orchestra.run.smithi079.stdout:(9/32): python3-cephfs-19.0.0-3227.g424d8c43.el 2.3 MB/s | 159 kB 00:00 2024-04-24T01:52:13.022 INFO:teuthology.orchestra.run.smithi079.stdout:(10/32): python3-rbd-19.0.0-3227.g424d8c43.el9. 4.9 MB/s | 293 kB 00:00 2024-04-24T01:52:13.056 INFO:teuthology.orchestra.run.smithi079.stdout:(11/32): python3-rados-19.0.0-3227.g424d8c43.el 2.6 MB/s | 315 kB 00:00 2024-04-24T01:52:13.081 INFO:teuthology.orchestra.run.smithi079.stdout:(12/32): python3-rgw-19.0.0-3227.g424d8c43.el9. 1.7 MB/s | 99 kB 00:00 2024-04-24T01:52:13.190 INFO:teuthology.orchestra.run.smithi079.stdout:(13/32): librgw2-19.0.0-3227.g424d8c43.el9.x86_ 8.7 MB/s | 5.2 MB 00:00 2024-04-24T01:52:13.298 INFO:teuthology.orchestra.run.smithi079.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 441 kB/s | 107 kB 00:00 2024-04-24T01:52:13.323 INFO:teuthology.orchestra.run.smithi079.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 339 kB/s | 45 kB 00:00 2024-04-24T01:52:13.349 INFO:teuthology.orchestra.run.smithi079.stdout:(16/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 599 kB/s | 160 kB 00:00 2024-04-24T01:52:13.399 INFO:teuthology.orchestra.run.smithi079.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 847 kB/s | 42 kB 00:00 2024-04-24T01:52:13.491 INFO:teuthology.orchestra.run.smithi079.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.7 MB/s | 292 kB 00:00 2024-04-24T01:52:13.516 INFO:teuthology.orchestra.run.smithi079.stdout:(19/32): qatlib-24.02.0-1.el9.x86_64.rpm 1.8 MB/s | 221 kB 00:00 2024-04-24T01:52:13.550 INFO:teuthology.orchestra.run.smithi079.stdout:(20/32): librdkafka-1.6.1-102.el9.x86_64.rpm 2.6 MB/s | 662 kB 00:00 2024-04-24T01:52:13.575 INFO:teuthology.orchestra.run.smithi079.stdout:(21/32): qatlib-service-24.02.0-1.el9.x86_64.rp 429 kB/s | 36 kB 00:00 2024-04-24T01:52:13.600 INFO:teuthology.orchestra.run.smithi079.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 566 kB/s | 47 kB 00:00 2024-04-24T01:52:13.875 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-04-24T01:52:13.875 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-base-2:19.0.0-3227.g424d8c43.el9.x86_64 1/34 2024-04-24T01:52:13.875 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-common-2:19.0.0-3227.g424d8c43.el9.x86_64 2/34 2024-04-24T01:52:13.875 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-radosgw-2:19.0.0-3227.g424d8c43.el9.x86_64 3/34 2024-04-24T01:52:13.875 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-selinux-2:19.0.0-3227.g424d8c43.el9.x86_64 4/34 2024-04-24T01:52:13.875 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libcephfs2-2:19.0.0-3227.g424d8c43.el9.x86_64 5/34 2024-04-24T01:52:13.875 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libradosstriper1-2:19.0.0-3227.g424d8c43.el9.x86_6 6/34 2024-04-24T01:52:13.876 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librgw2-2:19.0.0-3227.g424d8c43.el9.x86_64 7/34 2024-04-24T01:52:13.876 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3227.g424d8c43.el9. 8/34 2024-04-24T01:52:13.876 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-ceph-common-2:19.0.0-3227.g424d8c43.el9.x8 9/34 2024-04-24T01:52:13.876 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cephfs-2:19.0.0-3227.g424d8c43.el9.x86_64 10/34 2024-04-24T01:52:13.876 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-rados-2:19.0.0-3227.g424d8c43.el9.x86_64 11/34 2024-04-24T01:52:13.876 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-rbd-2:19.0.0-3227.g424d8c43.el9.x86_64 12/34 2024-04-24T01:52:13.876 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-rgw-2:19.0.0-3227.g424d8c43.el9.x86_64 13/34 2024-04-24T01:52:13.876 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-04-24T01:52:13.876 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-04-24T01:52:13.876 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-04-24T01:52:13.876 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-04-24T01:52:13.876 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-04-24T01:52:13.876 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-04-24T01:52:13.876 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-04-24T01:52:13.876 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-04-24T01:52:13.877 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-04-24T01:52:13.877 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-04-24T01:52:13.877 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-04-24T01:52:13.877 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-04-24T01:52:13.877 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-04-24T01:52:13.877 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-04-24T01:52:13.877 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-04-24T01:52:13.877 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-04-24T01:52:13.877 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-04-24T01:52:13.877 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librados2-2:19.0.0-3227.g424d8c43.el9.x86_64 31/34 2024-04-24T01:52:13.877 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-04-24T01:52:13.877 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librbd1-2:19.0.0-3227.g424d8c43.el9.x86_64 33/34 2024-04-24T01:52:13.878 INFO:teuthology.orchestra.run.smithi079.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 94 kB/s | 26 kB 00:00 2024-04-24T01:52:14.126 INFO:teuthology.orchestra.run.smithi079.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 194 kB/s | 49 kB 00:00 2024-04-24T01:52:14.193 INFO:teuthology.orchestra.run.smithi079.stdout:(25/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 481 kB/s | 309 kB 00:00 2024-04-24T01:52:14.376 INFO:teuthology.orchestra.run.smithi079.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 269 kB/s | 67 kB 00:00 2024-04-24T01:52:14.593 INFO:teuthology.orchestra.run.smithi079.stdout:(27/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.0 MB/s | 838 kB 00:00 2024-04-24T01:52:14.619 INFO:teuthology.orchestra.run.smithi079.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 791 kB/s | 192 kB 00:00 2024-04-24T01:52:14.657 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-04-24T01:52:14.657 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:52:14.657 INFO:teuthology.orchestra.run.smithi103.stdout:Upgraded: 2024-04-24T01:52:14.657 INFO:teuthology.orchestra.run.smithi103.stdout: librados2-2:19.0.0-3227.g424d8c43.el9.x86_64 2024-04-24T01:52:14.657 INFO:teuthology.orchestra.run.smithi103.stdout: librbd1-2:19.0.0-3227.g424d8c43.el9.x86_64 2024-04-24T01:52:14.657 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-04-24T01:52:14.657 INFO:teuthology.orchestra.run.smithi103.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-24T01:52:14.657 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-base-2:19.0.0-3227.g424d8c43.el9.x86_64 2024-04-24T01:52:14.657 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-common-2:19.0.0-3227.g424d8c43.el9.x86_64 2024-04-24T01:52:14.657 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-radosgw-2:19.0.0-3227.g424d8c43.el9.x86_64 2024-04-24T01:52:14.658 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-selinux-2:19.0.0-3227.g424d8c43.el9.x86_64 2024-04-24T01:52:14.658 INFO:teuthology.orchestra.run.smithi103.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-24T01:52:14.658 INFO:teuthology.orchestra.run.smithi103.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-24T01:52:14.658 INFO:teuthology.orchestra.run.smithi103.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-24T01:52:14.658 INFO:teuthology.orchestra.run.smithi103.stdout: libcephfs2-2:19.0.0-3227.g424d8c43.el9.x86_64 2024-04-24T01:52:14.658 INFO:teuthology.orchestra.run.smithi103.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-24T01:52:14.658 INFO:teuthology.orchestra.run.smithi103.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-24T01:52:14.658 INFO:teuthology.orchestra.run.smithi103.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-24T01:52:14.658 INFO:teuthology.orchestra.run.smithi103.stdout: libradosstriper1-2:19.0.0-3227.g424d8c43.el9.x86_64 2024-04-24T01:52:14.658 INFO:teuthology.orchestra.run.smithi103.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-24T01:52:14.658 INFO:teuthology.orchestra.run.smithi103.stdout: librgw2-2:19.0.0-3227.g424d8c43.el9.x86_64 2024-04-24T01:52:14.658 INFO:teuthology.orchestra.run.smithi103.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-24T01:52:14.658 INFO:teuthology.orchestra.run.smithi103.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-24T01:52:14.658 INFO:teuthology.orchestra.run.smithi103.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-24T01:52:14.659 INFO:teuthology.orchestra.run.smithi103.stdout: python3-ceph-argparse-2:19.0.0-3227.g424d8c43.el9.x86_64 2024-04-24T01:52:14.659 INFO:teuthology.orchestra.run.smithi103.stdout: python3-ceph-common-2:19.0.0-3227.g424d8c43.el9.x86_64 2024-04-24T01:52:14.659 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cephfs-2:19.0.0-3227.g424d8c43.el9.x86_64 2024-04-24T01:52:14.659 INFO:teuthology.orchestra.run.smithi103.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-24T01:52:14.659 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rados-2:19.0.0-3227.g424d8c43.el9.x86_64 2024-04-24T01:52:14.659 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rbd-2:19.0.0-3227.g424d8c43.el9.x86_64 2024-04-24T01:52:14.659 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rgw-2:19.0.0-3227.g424d8c43.el9.x86_64 2024-04-24T01:52:14.659 INFO:teuthology.orchestra.run.smithi103.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-24T01:52:14.659 INFO:teuthology.orchestra.run.smithi103.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-24T01:52:14.659 INFO:teuthology.orchestra.run.smithi103.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-24T01:52:14.659 INFO:teuthology.orchestra.run.smithi103.stdout: re2-1:20211101-3.el9.x86_64 2024-04-24T01:52:14.659 INFO:teuthology.orchestra.run.smithi103.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-24T01:52:14.659 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:52:14.659 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-04-24T01:52:14.928 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-test 2024-04-24T01:52:15.102 INFO:teuthology.orchestra.run.smithi079.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 3.1 MB/s | 1.6 MB 00:00 2024-04-24T01:52:15.311 INFO:teuthology.orchestra.run.smithi079.stdout:(30/32): librados2-19.0.0-3227.g424d8c43.el9.x8 4.9 MB/s | 3.4 MB 00:00 2024-04-24T01:52:15.369 INFO:teuthology.orchestra.run.smithi079.stdout:(31/32): libarrow-9.0.0-10.el9.x86_64.rpm 2.5 MB/s | 4.4 MB 00:01 2024-04-24T01:52:15.468 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:00:29 ago on Wed 24 Apr 2024 01:51:46 AM UTC. 2024-04-24T01:52:15.520 INFO:teuthology.orchestra.run.smithi079.stdout:(32/32): librbd1-19.0.0-3227.g424d8c43.el9.x86_ 8.8 MB/s | 3.7 MB 00:00 2024-04-24T01:52:15.522 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-04-24T01:52:15.522 INFO:teuthology.orchestra.run.smithi079.stdout:Total 11 MB/s | 59 MB 00:05 2024-04-24T01:52:15.591 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-04-24T01:52:15.592 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-24T01:52:15.593 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repository Size 2024-04-24T01:52:15.593 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-24T01:52:15.593 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-04-24T01:52:15.593 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-test x86_64 2:19.0.0-3227.g424d8c43.el9 ceph 84 M 2024-04-24T01:52:15.593 INFO:teuthology.orchestra.run.smithi103.stdout:Installing dependencies: 2024-04-24T01:52:15.593 INFO:teuthology.orchestra.run.smithi103.stdout: libasan8 x86_64 13.2.1-6.3.el9 appstream 469 k 2024-04-24T01:52:15.593 INFO:teuthology.orchestra.run.smithi103.stdout: libcephsqlite x86_64 2:19.0.0-3227.g424d8c43.el9 ceph 173 k 2024-04-24T01:52:15.593 INFO:teuthology.orchestra.run.smithi103.stdout: libubsan x86_64 11.4.1-3.el9 appstream 187 k 2024-04-24T01:52:15.593 INFO:teuthology.orchestra.run.smithi103.stdout: lksctp-tools x86_64 1.0.19-2.el9 baseos 94 k 2024-04-24T01:52:15.593 INFO:teuthology.orchestra.run.smithi103.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-24T01:52:15.594 INFO:teuthology.orchestra.run.smithi103.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-24T01:52:15.594 INFO:teuthology.orchestra.run.smithi103.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-24T01:52:15.594 INFO:teuthology.orchestra.run.smithi103.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2024-04-24T01:52:15.594 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:52:15.594 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-04-24T01:52:15.594 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-24T01:52:15.594 INFO:teuthology.orchestra.run.smithi103.stdout:Install 9 Packages 2024-04-24T01:52:15.594 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:52:15.595 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 87 M 2024-04-24T01:52:15.595 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 813 M 2024-04-24T01:52:15.595 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-04-24T01:52:15.627 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-04-24T01:52:15.664 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-04-24T01:52:15.664 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-04-24T01:52:16.124 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-04-24T01:52:16.125 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-04-24T01:52:16.323 INFO:teuthology.orchestra.run.smithi103.stdout:(1/9): lksctp-tools-1.0.19-2.el9.x86_64.rpm 670 kB/s | 94 kB 00:00 2024-04-24T01:52:16.583 INFO:teuthology.orchestra.run.smithi103.stdout:(2/9): libcephsqlite-19.0.0-3227.g424d8c43.el9. 433 kB/s | 173 kB 00:00 2024-04-24T01:52:17.488 INFO:teuthology.orchestra.run.smithi103.stdout:(3/9): libubsan-11.4.1-3.el9.x86_64.rpm 213 kB/s | 187 kB 00:00 2024-04-24T01:52:17.589 INFO:teuthology.orchestra.run.smithi103.stdout:(4/9): libasan8-13.2.1-6.3.el9.x86_64.rpm 370 kB/s | 469 kB 00:01 2024-04-24T01:52:17.599 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-04-24T01:52:18.117 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-04-24T01:52:18.172 INFO:teuthology.orchestra.run.smithi079.stdout: Upgrading : librados2-2:19.0.0-3227.g424d8c43.el9.x86_64 2/34 2024-04-24T01:52:18.263 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: librados2-2:19.0.0-3227.g424d8c43.el9.x86_64 2/34 2024-04-24T01:52:18.351 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-rados-2:19.0.0-3227.g424d8c43.el9.x86_64 3/34 2024-04-24T01:52:18.382 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libcephfs2-2:19.0.0-3227.g424d8c43.el9.x86_64 4/34 2024-04-24T01:52:18.444 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: libcephfs2-2:19.0.0-3227.g424d8c43.el9.x86_64 4/34 2024-04-24T01:52:18.481 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : liboath-2.6.7-2.el9.x86_64 5/34 2024-04-24T01:52:18.511 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/34 2024-04-24T01:52:18.536 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/34 2024-04-24T01:52:18.556 INFO:teuthology.orchestra.run.smithi103.stdout:(5/9): socat-1.7.4.1-5.el9.x86_64.rpm 315 kB/s | 305 kB 00:00 2024-04-24T01:52:18.559 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 8/34 2024-04-24T01:52:18.579 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 8/34 2024-04-24T01:52:18.776 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 9/34 2024-04-24T01:52:18.808 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 9/34 2024-04-24T01:52:19.165 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 10/34 2024-04-24T01:52:19.221 INFO:teuthology.orchestra.run.smithi079.stdout: Upgrading : librbd1-2:19.0.0-3227.g424d8c43.el9.x86_64 11/34 2024-04-24T01:52:19.285 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: librbd1-2:19.0.0-3227.g424d8c43.el9.x86_64 11/34 2024-04-24T01:52:19.331 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-ceph-argparse-2:19.0.0-3227.g424d8c43.el9. 12/34 2024-04-24T01:52:19.381 INFO:teuthology.orchestra.run.smithi103.stdout:(6/9): xmlstarlet-1.6.1-20.el9.x86_64.rpm 77 kB/s | 64 kB 00:00 2024-04-24T01:52:19.386 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-cephfs-2:19.0.0-3227.g424d8c43.el9.x86_64 13/34 2024-04-24T01:52:19.419 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-rbd-2:19.0.0-3227.g424d8c43.el9.x86_64 14/34 2024-04-24T01:52:19.499 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 15/34 2024-04-24T01:52:19.524 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libradosstriper1-2:19.0.0-3227.g424d8c43.el9.x86_6 16/34 2024-04-24T01:52:19.565 INFO:teuthology.orchestra.run.smithi103.stdout:(7/9): protobuf-3.14.0-13.el9.x86_64.rpm 498 kB/s | 1.0 MB 00:02 2024-04-24T01:52:19.612 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3227.g424d8c43.el9.x86_6 16/34 2024-04-24T01:52:19.639 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : thrift-0.15.0-2.el9.x86_64 17/34 2024-04-24T01:52:19.749 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : re2-1:20211101-3.el9.x86_64 18/34 2024-04-24T01:52:19.781 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 19/34 2024-04-24T01:52:19.821 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 20/34 2024-04-24T01:52:19.847 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 21/34 2024-04-24T01:52:19.878 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 22/34 2024-04-24T01:52:19.907 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 23/34 2024-04-24T01:52:19.943 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-04-24T01:52:20.149 INFO:teuthology.orchestra.run.smithi103.stdout:(8/9): ceph-test-19.0.0-3227.g424d8c43.el9.x86_ 21 MB/s | 84 MB 00:03 2024-04-24T01:52:20.282 INFO:teuthology.orchestra.run.smithi103.stdout:(9/9): yaml-cpp-0.6.3-5.el9.x86_64.rpm 135 kB/s | 122 kB 00:00 2024-04-24T01:52:20.286 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-04-24T01:52:20.286 INFO:teuthology.orchestra.run.smithi103.stdout:Total 18 MB/s | 87 MB 00:04 2024-04-24T01:52:20.367 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-04-24T01:52:20.390 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-04-24T01:52:20.390 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-04-24T01:52:20.523 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-04-24T01:52:20.547 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : librgw2-2:19.0.0-3227.g424d8c43.el9.x86_64 26/34 2024-04-24T01:52:20.602 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: librgw2-2:19.0.0-3227.g424d8c43.el9.x86_64 26/34 2024-04-24T01:52:20.678 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-rgw-2:19.0.0-3227.g424d8c43.el9.x86_64 27/34 2024-04-24T01:52:20.708 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-ceph-common-2:19.0.0-3227.g424d8c43.el9.x8 28/34 2024-04-24T01:52:20.987 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-04-24T01:52:20.988 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-04-24T01:52:21.904 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-04-24T01:52:21.998 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/9 2024-04-24T01:52:22.041 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 2/9 2024-04-24T01:52:22.086 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 3/9 2024-04-24T01:52:22.122 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 4/9 2024-04-24T01:52:22.151 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libubsan-11.4.1-3.el9.x86_64 5/9 2024-04-24T01:52:22.177 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libasan8-13.2.1-6.3.el9.x86_64 6/9 2024-04-24T01:52:22.246 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: libasan8-13.2.1-6.3.el9.x86_64 6/9 2024-04-24T01:52:22.286 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 7/9 2024-04-24T01:52:22.312 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libcephsqlite-2:19.0.0-3227.g424d8c43.el9.x86_64 8/9 2024-04-24T01:52:23.144 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-common-2:19.0.0-3227.g424d8c43.el9.x86_64 29/34 2024-04-24T01:52:23.263 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-common-2:19.0.0-3227.g424d8c43.el9.x86_64 29/34 2024-04-24T01:52:23.921 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-common-2:19.0.0-3227.g424d8c43.el9.x86_64 29/34 2024-04-24T01:52:23.982 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-base-2:19.0.0-3227.g424d8c43.el9.x86_64 30/34 2024-04-24T01:52:24.078 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-base-2:19.0.0-3227.g424d8c43.el9.x86_64 30/34 2024-04-24T01:52:24.079 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-24T01:52:24.079 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-24T01:52:24.079 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:52:24.125 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-selinux-2:19.0.0-3227.g424d8c43.el9.x86_64 31/34 2024-04-24T01:52:33.609 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3227.g424d8c43.el9.x86_64 8/9 2024-04-24T01:52:33.684 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-test-2:19.0.0-3227.g424d8c43.el9.x86_64 9/9 2024-04-24T01:52:33.966 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3227.g424d8c43.el9.x86_64 31/34 2024-04-24T01:52:33.966 INFO:teuthology.orchestra.run.smithi079.stdout:skipping the directory /sys 2024-04-24T01:52:33.967 INFO:teuthology.orchestra.run.smithi079.stdout:skipping the directory /proc 2024-04-24T01:52:33.967 INFO:teuthology.orchestra.run.smithi079.stdout:skipping the directory /mnt 2024-04-24T01:52:33.967 INFO:teuthology.orchestra.run.smithi079.stdout:skipping the directory /var/tmp 2024-04-24T01:52:33.967 INFO:teuthology.orchestra.run.smithi079.stdout:skipping the directory /home 2024-04-24T01:52:33.967 INFO:teuthology.orchestra.run.smithi079.stdout:skipping the directory /root 2024-04-24T01:52:33.967 INFO:teuthology.orchestra.run.smithi079.stdout:skipping the directory /tmp 2024-04-24T01:52:33.967 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:52:35.083 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-radosgw-2:19.0.0-3227.g424d8c43.el9.x86_64 32/34 2024-04-24T01:52:35.107 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3227.g424d8c43.el9.x86_64 32/34 2024-04-24T01:52:35.107 INFO:teuthology.orchestra.run.smithi079.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T01:52:35.107 INFO:teuthology.orchestra.run.smithi079.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-24T01:52:35.107 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-24T01:52:35.108 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-24T01:52:35.108 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:52:35.109 INFO:teuthology.orchestra.run.smithi079.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-04-24T01:52:35.185 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-04-24T01:52:35.185 INFO:teuthology.orchestra.run.smithi079.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-04-24T01:52:36.485 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-04-24T01:52:36.486 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-base-2:19.0.0-3227.g424d8c43.el9.x86_64 1/34 2024-04-24T01:52:36.486 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-common-2:19.0.0-3227.g424d8c43.el9.x86_64 2/34 2024-04-24T01:52:36.486 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-radosgw-2:19.0.0-3227.g424d8c43.el9.x86_64 3/34 2024-04-24T01:52:36.486 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-selinux-2:19.0.0-3227.g424d8c43.el9.x86_64 4/34 2024-04-24T01:52:36.486 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libcephfs2-2:19.0.0-3227.g424d8c43.el9.x86_64 5/34 2024-04-24T01:52:36.486 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libradosstriper1-2:19.0.0-3227.g424d8c43.el9.x86_6 6/34 2024-04-24T01:52:36.486 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librgw2-2:19.0.0-3227.g424d8c43.el9.x86_64 7/34 2024-04-24T01:52:36.486 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3227.g424d8c43.el9. 8/34 2024-04-24T01:52:36.486 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-ceph-common-2:19.0.0-3227.g424d8c43.el9.x8 9/34 2024-04-24T01:52:36.486 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-cephfs-2:19.0.0-3227.g424d8c43.el9.x86_64 10/34 2024-04-24T01:52:36.486 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-rados-2:19.0.0-3227.g424d8c43.el9.x86_64 11/34 2024-04-24T01:52:36.486 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-rbd-2:19.0.0-3227.g424d8c43.el9.x86_64 12/34 2024-04-24T01:52:36.487 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-rgw-2:19.0.0-3227.g424d8c43.el9.x86_64 13/34 2024-04-24T01:52:36.487 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-04-24T01:52:36.487 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-04-24T01:52:36.487 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-04-24T01:52:36.487 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-04-24T01:52:36.487 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-04-24T01:52:36.487 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-04-24T01:52:36.487 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-04-24T01:52:36.487 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-04-24T01:52:36.487 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-04-24T01:52:36.487 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-04-24T01:52:36.487 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-04-24T01:52:36.487 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-04-24T01:52:36.487 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-04-24T01:52:36.487 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-04-24T01:52:36.487 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-04-24T01:52:36.488 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-04-24T01:52:36.488 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-04-24T01:52:36.488 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librados2-2:19.0.0-3227.g424d8c43.el9.x86_64 31/34 2024-04-24T01:52:36.488 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-04-24T01:52:36.488 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librbd1-2:19.0.0-3227.g424d8c43.el9.x86_64 33/34 2024-04-24T01:52:37.250 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-04-24T01:52:37.250 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:52:37.250 INFO:teuthology.orchestra.run.smithi079.stdout:Upgraded: 2024-04-24T01:52:37.250 INFO:teuthology.orchestra.run.smithi079.stdout: librados2-2:19.0.0-3227.g424d8c43.el9.x86_64 2024-04-24T01:52:37.250 INFO:teuthology.orchestra.run.smithi079.stdout: librbd1-2:19.0.0-3227.g424d8c43.el9.x86_64 2024-04-24T01:52:37.250 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-04-24T01:52:37.250 INFO:teuthology.orchestra.run.smithi079.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-24T01:52:37.250 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-base-2:19.0.0-3227.g424d8c43.el9.x86_64 2024-04-24T01:52:37.250 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-common-2:19.0.0-3227.g424d8c43.el9.x86_64 2024-04-24T01:52:37.250 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-radosgw-2:19.0.0-3227.g424d8c43.el9.x86_64 2024-04-24T01:52:37.251 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-selinux-2:19.0.0-3227.g424d8c43.el9.x86_64 2024-04-24T01:52:37.251 INFO:teuthology.orchestra.run.smithi079.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-24T01:52:37.251 INFO:teuthology.orchestra.run.smithi079.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-24T01:52:37.251 INFO:teuthology.orchestra.run.smithi079.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-24T01:52:37.251 INFO:teuthology.orchestra.run.smithi079.stdout: libcephfs2-2:19.0.0-3227.g424d8c43.el9.x86_64 2024-04-24T01:52:37.251 INFO:teuthology.orchestra.run.smithi079.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-24T01:52:37.251 INFO:teuthology.orchestra.run.smithi079.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-24T01:52:37.251 INFO:teuthology.orchestra.run.smithi079.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-24T01:52:37.251 INFO:teuthology.orchestra.run.smithi079.stdout: libradosstriper1-2:19.0.0-3227.g424d8c43.el9.x86_64 2024-04-24T01:52:37.251 INFO:teuthology.orchestra.run.smithi079.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-24T01:52:37.251 INFO:teuthology.orchestra.run.smithi079.stdout: librgw2-2:19.0.0-3227.g424d8c43.el9.x86_64 2024-04-24T01:52:37.251 INFO:teuthology.orchestra.run.smithi079.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-24T01:52:37.251 INFO:teuthology.orchestra.run.smithi079.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-24T01:52:37.251 INFO:teuthology.orchestra.run.smithi079.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-24T01:52:37.252 INFO:teuthology.orchestra.run.smithi079.stdout: python3-ceph-argparse-2:19.0.0-3227.g424d8c43.el9.x86_64 2024-04-24T01:52:37.252 INFO:teuthology.orchestra.run.smithi079.stdout: python3-ceph-common-2:19.0.0-3227.g424d8c43.el9.x86_64 2024-04-24T01:52:37.252 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cephfs-2:19.0.0-3227.g424d8c43.el9.x86_64 2024-04-24T01:52:37.252 INFO:teuthology.orchestra.run.smithi079.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-24T01:52:37.252 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rados-2:19.0.0-3227.g424d8c43.el9.x86_64 2024-04-24T01:52:37.252 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rbd-2:19.0.0-3227.g424d8c43.el9.x86_64 2024-04-24T01:52:37.252 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rgw-2:19.0.0-3227.g424d8c43.el9.x86_64 2024-04-24T01:52:37.252 INFO:teuthology.orchestra.run.smithi079.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-24T01:52:37.252 INFO:teuthology.orchestra.run.smithi079.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-24T01:52:37.252 INFO:teuthology.orchestra.run.smithi079.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-24T01:52:37.252 INFO:teuthology.orchestra.run.smithi079.stdout: re2-1:20211101-3.el9.x86_64 2024-04-24T01:52:37.252 INFO:teuthology.orchestra.run.smithi079.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-24T01:52:37.252 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:52:37.252 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-04-24T01:52:37.503 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-test 2024-04-24T01:52:38.038 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:00:31 ago on Wed 24 Apr 2024 01:52:07 AM UTC. 2024-04-24T01:52:38.158 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-04-24T01:52:38.159 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-04-24T01:52:38.159 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2024-04-24T01:52:38.160 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-04-24T01:52:38.160 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-04-24T01:52:38.160 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-test x86_64 2:19.0.0-3227.g424d8c43.el9 ceph 84 M 2024-04-24T01:52:38.160 INFO:teuthology.orchestra.run.smithi079.stdout:Installing dependencies: 2024-04-24T01:52:38.160 INFO:teuthology.orchestra.run.smithi079.stdout: libasan8 x86_64 13.2.1-6.3.el9 appstream 469 k 2024-04-24T01:52:38.160 INFO:teuthology.orchestra.run.smithi079.stdout: libcephsqlite x86_64 2:19.0.0-3227.g424d8c43.el9 ceph 173 k 2024-04-24T01:52:38.160 INFO:teuthology.orchestra.run.smithi079.stdout: libubsan x86_64 11.4.1-3.el9 appstream 187 k 2024-04-24T01:52:38.160 INFO:teuthology.orchestra.run.smithi079.stdout: lksctp-tools x86_64 1.0.19-2.el9 baseos 94 k 2024-04-24T01:52:38.160 INFO:teuthology.orchestra.run.smithi079.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-24T01:52:38.160 INFO:teuthology.orchestra.run.smithi079.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-24T01:52:38.160 INFO:teuthology.orchestra.run.smithi079.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-24T01:52:38.160 INFO:teuthology.orchestra.run.smithi079.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2024-04-24T01:52:38.160 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:52:38.160 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-04-24T01:52:38.160 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-04-24T01:52:38.161 INFO:teuthology.orchestra.run.smithi079.stdout:Install 9 Packages 2024-04-24T01:52:38.161 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:52:38.161 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 87 M 2024-04-24T01:52:38.161 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 813 M 2024-04-24T01:52:38.161 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-04-24T01:52:38.559 INFO:teuthology.orchestra.run.smithi079.stdout:(1/9): lksctp-tools-1.0.19-2.el9.x86_64.rpm 681 kB/s | 94 kB 00:00 2024-04-24T01:52:38.673 INFO:teuthology.orchestra.run.smithi079.stdout:(2/9): libcephsqlite-19.0.0-3227.g424d8c43.el9. 686 kB/s | 173 kB 00:00 2024-04-24T01:52:38.752 INFO:teuthology.orchestra.run.smithi079.stdout:(3/9): libasan8-13.2.1-6.3.el9.x86_64.rpm 2.4 MB/s | 469 kB 00:00 2024-04-24T01:52:38.812 INFO:teuthology.orchestra.run.smithi079.stdout:(4/9): libubsan-11.4.1-3.el9.x86_64.rpm 1.3 MB/s | 187 kB 00:00 2024-04-24T01:52:38.870 INFO:teuthology.orchestra.run.smithi079.stdout:(5/9): protobuf-3.14.0-13.el9.x86_64.rpm 8.6 MB/s | 1.0 MB 00:00 2024-04-24T01:52:38.929 INFO:teuthology.orchestra.run.smithi079.stdout:(6/9): socat-1.7.4.1-5.el9.x86_64.rpm 2.5 MB/s | 305 kB 00:00 2024-04-24T01:52:38.965 INFO:teuthology.orchestra.run.smithi079.stdout:(7/9): xmlstarlet-1.6.1-20.el9.x86_64.rpm 676 kB/s | 64 kB 00:00 2024-04-24T01:52:39.611 INFO:teuthology.orchestra.run.smithi079.stdout:(8/9): yaml-cpp-0.6.3-5.el9.x86_64.rpm 179 kB/s | 122 kB 00:00 2024-04-24T01:52:40.339 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-test-2:19.0.0-3227.g424d8c43.el9.x86_64 9/9 2024-04-24T01:52:40.340 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-test-2:19.0.0-3227.g424d8c43.el9.x86_64 1/9 2024-04-24T01:52:40.340 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libcephsqlite-2:19.0.0-3227.g424d8c43.el9.x86_64 2/9 2024-04-24T01:52:40.340 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 3/9 2024-04-24T01:52:40.340 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libasan8-13.2.1-6.3.el9.x86_64 4/9 2024-04-24T01:52:40.340 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libubsan-11.4.1-3.el9.x86_64 5/9 2024-04-24T01:52:40.340 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 6/9 2024-04-24T01:52:40.341 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 7/9 2024-04-24T01:52:40.341 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 8/9 2024-04-24T01:52:40.686 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 9/9 2024-04-24T01:52:40.687 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:52:40.687 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-04-24T01:52:40.687 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-test-2:19.0.0-3227.g424d8c43.el9.x86_64 2024-04-24T01:52:40.687 INFO:teuthology.orchestra.run.smithi103.stdout: libasan8-13.2.1-6.3.el9.x86_64 2024-04-24T01:52:40.687 INFO:teuthology.orchestra.run.smithi103.stdout: libcephsqlite-2:19.0.0-3227.g424d8c43.el9.x86_64 2024-04-24T01:52:40.687 INFO:teuthology.orchestra.run.smithi103.stdout: libubsan-11.4.1-3.el9.x86_64 2024-04-24T01:52:40.687 INFO:teuthology.orchestra.run.smithi103.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2024-04-24T01:52:40.687 INFO:teuthology.orchestra.run.smithi103.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-24T01:52:40.687 INFO:teuthology.orchestra.run.smithi103.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-24T01:52:40.687 INFO:teuthology.orchestra.run.smithi103.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-24T01:52:40.687 INFO:teuthology.orchestra.run.smithi103.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2024-04-24T01:52:40.687 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:52:40.687 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-04-24T01:52:40.965 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph 2024-04-24T01:52:41.486 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:00:55 ago on Wed 24 Apr 2024 01:51:46 AM UTC. 2024-04-24T01:52:41.616 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-04-24T01:52:41.618 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-24T01:52:41.618 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repository Size 2024-04-24T01:52:41.618 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-24T01:52:41.618 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-04-24T01:52:41.618 INFO:teuthology.orchestra.run.smithi103.stdout: ceph x86_64 2:19.0.0-3227.g424d8c43.el9 ceph 6.5 k 2024-04-24T01:52:41.619 INFO:teuthology.orchestra.run.smithi103.stdout:Installing dependencies: 2024-04-24T01:52:41.619 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mds x86_64 2:19.0.0-3227.g424d8c43.el9 ceph 2.2 M 2024-04-24T01:52:41.619 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr x86_64 2:19.0.0-3227.g424d8c43.el9 ceph 1.6 M 2024-04-24T01:52:41.619 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3227.g424d8c43.el9 ceph-noarch 249 k 2024-04-24T01:52:41.619 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mon x86_64 2:19.0.0-3227.g424d8c43.el9 ceph 5.0 M 2024-04-24T01:52:41.619 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-osd x86_64 2:19.0.0-3227.g424d8c43.el9 ceph 71 M 2024-04-24T01:52:41.619 INFO:teuthology.orchestra.run.smithi103.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-24T01:52:41.619 INFO:teuthology.orchestra.run.smithi103.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-24T01:52:41.619 INFO:teuthology.orchestra.run.smithi103.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-24T01:52:41.619 INFO:teuthology.orchestra.run.smithi103.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-24T01:52:41.619 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-24T01:52:41.619 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-24T01:52:41.620 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-24T01:52:41.620 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-24T01:52:41.620 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-24T01:52:41.620 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-24T01:52:41.620 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-collections 2024-04-24T01:52:41.620 INFO:teuthology.orchestra.run.smithi103.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-04-24T01:52:41.620 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-24T01:52:41.620 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-24T01:52:41.620 INFO:teuthology.orchestra.run.smithi103.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-24T01:52:41.620 INFO:teuthology.orchestra.run.smithi103.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-24T01:52:41.620 INFO:teuthology.orchestra.run.smithi103.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-24T01:52:41.620 INFO:teuthology.orchestra.run.smithi103.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-24T01:52:41.621 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-24T01:52:41.621 INFO:teuthology.orchestra.run.smithi103.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-24T01:52:41.621 INFO:teuthology.orchestra.run.smithi103.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-24T01:52:41.621 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-24T01:52:41.621 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-24T01:52:41.621 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-24T01:52:41.621 INFO:teuthology.orchestra.run.smithi103.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-24T01:52:41.621 INFO:teuthology.orchestra.run.smithi103.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-24T01:52:41.621 INFO:teuthology.orchestra.run.smithi103.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-24T01:52:41.621 INFO:teuthology.orchestra.run.smithi103.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-24T01:52:41.621 INFO:teuthology.orchestra.run.smithi103.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-24T01:52:41.621 INFO:teuthology.orchestra.run.smithi103.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-24T01:52:41.621 INFO:teuthology.orchestra.run.smithi103.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-24T01:52:41.621 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:52:41.622 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-04-24T01:52:41.622 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-24T01:52:41.622 INFO:teuthology.orchestra.run.smithi103.stdout:Install 36 Packages 2024-04-24T01:52:41.622 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:52:41.622 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 84 M 2024-04-24T01:52:41.623 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 922 M 2024-04-24T01:52:41.623 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-04-24T01:52:42.388 INFO:teuthology.orchestra.run.smithi079.stdout:(9/9): ceph-test-19.0.0-3227.g424d8c43.el9.x86_ 21 MB/s | 84 MB 00:03 2024-04-24T01:52:42.391 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-04-24T01:52:42.391 INFO:teuthology.orchestra.run.smithi079.stdout:Total 21 MB/s | 87 MB 00:04 2024-04-24T01:52:42.471 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-04-24T01:52:42.493 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-04-24T01:52:42.493 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-04-24T01:52:42.764 INFO:teuthology.orchestra.run.smithi103.stdout:(1/36): ceph-19.0.0-3227.g424d8c43.el9.x86_64.r 39 kB/s | 6.5 kB 00:00 2024-04-24T01:52:42.981 INFO:teuthology.orchestra.run.smithi103.stdout:(2/36): ceph-mds-19.0.0-3227.g424d8c43.el9.x86_ 5.7 MB/s | 2.2 MB 00:00 2024-04-24T01:52:43.090 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-04-24T01:52:43.090 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-04-24T01:52:43.373 INFO:teuthology.orchestra.run.smithi103.stdout:(3/36): ceph-mon-19.0.0-3227.g424d8c43.el9.x86_ 8.3 MB/s | 5.0 MB 00:00 2024-04-24T01:52:43.440 INFO:teuthology.orchestra.run.smithi103.stdout:(4/36): ceph-mgr-modules-core-19.0.0-3227.g424d 3.7 MB/s | 249 kB 00:00 2024-04-24T01:52:43.615 INFO:teuthology.orchestra.run.smithi103.stdout:(5/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.4 MB/s | 253 kB 00:00 2024-04-24T01:52:43.682 INFO:teuthology.orchestra.run.smithi103.stdout:(6/36): ceph-mgr-19.0.0-3227.g424d8c43.el9.x86_ 1.4 MB/s | 1.6 MB 00:01 2024-04-24T01:52:43.740 INFO:teuthology.orchestra.run.smithi103.stdout:(7/36): python3-cryptography-36.0.1-4.el9.x86_6 9.8 MB/s | 1.2 MB 00:00 2024-04-24T01:52:43.791 INFO:teuthology.orchestra.run.smithi103.stdout:(8/36): python3-pycparser-2.20-6.el9.noarch.rpm 2.6 MB/s | 135 kB 00:00 2024-04-24T01:52:43.816 INFO:teuthology.orchestra.run.smithi103.stdout:(9/36): python3-ply-3.11-14.el9.noarch.rpm 797 kB/s | 106 kB 00:00 2024-04-24T01:52:43.841 INFO:teuthology.orchestra.run.smithi103.stdout:(10/36): python3-requests-2.25.1-8.el9.noarch.r 2.4 MB/s | 125 kB 00:00 2024-04-24T01:52:43.884 INFO:teuthology.orchestra.run.smithi103.stdout:(11/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.1 MB/s | 215 kB 00:00 2024-04-24T01:52:43.969 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-04-24T01:52:44.017 INFO:teuthology.orchestra.run.smithi103.stdout:(12/36): lua-5.4.4-4.el9.x86_64.rpm 1.0 MB/s | 188 kB 00:00 2024-04-24T01:52:44.028 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/9 2024-04-24T01:52:44.042 INFO:teuthology.orchestra.run.smithi103.stdout:(13/36): python3-mako-1.1.4-6.el9.noarch.rpm 1.1 MB/s | 172 kB 00:00 2024-04-24T01:52:44.068 INFO:teuthology.orchestra.run.smithi103.stdout:(14/36): python3-markupsafe-1.1.1-12.el9.x86_64 690 kB/s | 35 kB 00:00 2024-04-24T01:52:44.072 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 2/9 2024-04-24T01:52:44.101 INFO:teuthology.orchestra.run.smithi103.stdout:(15/36): python3-pytz-2021.1-5.el9.noarch.rpm 869 kB/s | 51 kB 00:00 2024-04-24T01:52:44.117 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 3/9 2024-04-24T01:52:44.144 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 4/9 2024-04-24T01:52:44.174 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libubsan-11.4.1-3.el9.x86_64 5/9 2024-04-24T01:52:44.193 INFO:teuthology.orchestra.run.smithi103.stdout:(16/36): python3-toml-0.10.2-6.el9.noarch.rpm 332 kB/s | 42 kB 00:00 2024-04-24T01:52:44.199 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libasan8-13.2.1-6.3.el9.x86_64 6/9 2024-04-24T01:52:44.219 INFO:teuthology.orchestra.run.smithi103.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 190 kB/s | 22 kB 00:00 2024-04-24T01:52:44.268 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: libasan8-13.2.1-6.3.el9.x86_64 6/9 2024-04-24T01:52:44.308 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 7/9 2024-04-24T01:52:44.327 INFO:teuthology.orchestra.run.smithi103.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 402 kB/s | 43 kB 00:00 2024-04-24T01:52:44.335 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libcephsqlite-2:19.0.0-3227.g424d8c43.el9.x86_64 8/9 2024-04-24T01:52:44.353 INFO:teuthology.orchestra.run.smithi103.stdout:(19/36): luarocks-3.9.2-1.el9.noarch.rpm 950 kB/s | 151 kB 00:00 2024-04-24T01:52:44.428 INFO:teuthology.orchestra.run.smithi103.stdout:(20/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.7 MB/s | 172 kB 00:00 2024-04-24T01:52:44.453 INFO:teuthology.orchestra.run.smithi103.stdout:(21/36): python3-cherrypy-18.6.1-2.el9.noarch.r 3.5 MB/s | 358 kB 00:00 2024-04-24T01:52:44.478 INFO:teuthology.orchestra.run.smithi103.stdout:(22/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 213 kB/s | 11 kB 00:00 2024-04-24T01:52:44.504 INFO:teuthology.orchestra.run.smithi103.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 351 kB/s | 18 kB 00:00 2024-04-24T01:52:44.529 INFO:teuthology.orchestra.run.smithi103.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 460 kB/s | 23 kB 00:00 2024-04-24T01:52:44.554 INFO:teuthology.orchestra.run.smithi103.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 386 kB/s | 19 kB 00:00 2024-04-24T01:52:44.580 INFO:teuthology.orchestra.run.smithi103.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 390 kB/s | 20 kB 00:00 2024-04-24T01:52:44.605 INFO:teuthology.orchestra.run.smithi103.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 924 kB/s | 46 kB 00:00 2024-04-24T01:52:44.630 INFO:teuthology.orchestra.run.smithi103.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-04-24T01:52:44.689 INFO:teuthology.orchestra.run.smithi103.stdout:(29/36): python3-pecan-1.4.2-3.el9.noarch.rpm 3.2 MB/s | 272 kB 00:00 2024-04-24T01:52:44.706 INFO:teuthology.orchestra.run.smithi103.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 217 kB/s | 16 kB 00:00 2024-04-24T01:52:44.723 INFO:teuthology.orchestra.run.smithi103.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 2.6 MB/s | 90 kB 00:00 2024-04-24T01:52:44.756 INFO:teuthology.orchestra.run.smithi103.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 713 kB/s | 36 kB 00:00 2024-04-24T01:52:44.782 INFO:teuthology.orchestra.run.smithi103.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 3.8 MB/s | 230 kB 00:00 2024-04-24T01:52:44.815 INFO:teuthology.orchestra.run.smithi103.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 599 kB/s | 20 kB 00:00 2024-04-24T01:52:44.899 INFO:teuthology.orchestra.run.smithi103.stdout:(35/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.9 MB/s | 427 kB 00:00 2024-04-24T01:52:46.341 INFO:teuthology.orchestra.run.smithi103.stdout:(36/36): ceph-osd-19.0.0-3227.g424d8c43.el9.x86 21 MB/s | 71 MB 00:03 2024-04-24T01:52:46.344 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-04-24T01:52:46.344 INFO:teuthology.orchestra.run.smithi103.stdout:Total 18 MB/s | 84 MB 00:04 2024-04-24T01:52:46.482 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-04-24T01:52:46.523 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-04-24T01:52:46.524 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-04-24T01:52:47.140 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-04-24T01:52:47.141 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-04-24T01:52:48.704 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-04-24T01:52:48.769 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-04-24T01:52:48.810 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-04-24T01:52:48.838 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-04-24T01:52:48.909 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-04-24T01:52:48.970 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-04-24T01:52:49.003 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-04-24T01:52:49.096 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-04-24T01:52:49.182 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-04-24T01:52:49.222 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-04-24T01:52:49.258 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-04-24T01:52:49.304 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/36 2024-04-24T01:52:49.339 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/36 2024-04-24T01:52:49.372 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/36 2024-04-24T01:52:49.404 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/36 2024-04-24T01:52:49.440 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/36 2024-04-24T01:52:49.474 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/36 2024-04-24T01:52:49.512 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/36 2024-04-24T01:52:49.586 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 18/36 2024-04-24T01:52:49.733 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 19/36 2024-04-24T01:52:49.821 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 20/36 2024-04-24T01:52:49.873 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 21/36 2024-04-24T01:52:49.920 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 22/36 2024-04-24T01:52:49.970 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-ply-3.11-14.el9.noarch 23/36 2024-04-24T01:52:50.025 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 24/36 2024-04-24T01:52:50.213 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 25/36 2024-04-24T01:52:50.258 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 26/36 2024-04-24T01:52:50.337 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 27/36 2024-04-24T01:52:50.434 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 28/36 2024-04-24T01:52:50.474 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 29/36 2024-04-24T01:52:50.592 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 30/36 2024-04-24T01:52:50.783 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3227.g424d8c43.el9. 31/36 2024-04-24T01:52:50.810 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mgr-2:19.0.0-3227.g424d8c43.el9.x86_64 32/36 2024-04-24T01:52:50.835 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3227.g424d8c43.el9.x86_64 32/36 2024-04-24T01:52:50.835 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T01:52:50.835 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-24T01:52:50.835 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-24T01:52:50.835 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-24T01:52:50.835 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:52:55.642 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3227.g424d8c43.el9.x86_64 8/9 2024-04-24T01:52:55.780 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-test-2:19.0.0-3227.g424d8c43.el9.x86_64 9/9 2024-04-24T01:53:01.685 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-osd-2:19.0.0-3227.g424d8c43.el9.x86_64 33/36 2024-04-24T01:53:01.712 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-osd-2:19.0.0-3227.g424d8c43.el9.x86_64 33/36 2024-04-24T01:53:01.712 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T01:53:01.712 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-24T01:53:01.712 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-24T01:53:01.712 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-24T01:53:01.712 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:53:02.252 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mon-2:19.0.0-3227.g424d8c43.el9.x86_64 34/36 2024-04-24T01:53:02.275 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mon-2:19.0.0-3227.g424d8c43.el9.x86_64 34/36 2024-04-24T01:53:02.275 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T01:53:02.275 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-24T01:53:02.276 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-24T01:53:02.276 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-24T01:53:02.276 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:53:02.316 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-test-2:19.0.0-3227.g424d8c43.el9.x86_64 9/9 2024-04-24T01:53:02.317 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-test-2:19.0.0-3227.g424d8c43.el9.x86_64 1/9 2024-04-24T01:53:02.317 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libcephsqlite-2:19.0.0-3227.g424d8c43.el9.x86_64 2/9 2024-04-24T01:53:02.317 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 3/9 2024-04-24T01:53:02.317 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libasan8-13.2.1-6.3.el9.x86_64 4/9 2024-04-24T01:53:02.317 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libubsan-11.4.1-3.el9.x86_64 5/9 2024-04-24T01:53:02.317 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 6/9 2024-04-24T01:53:02.317 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 7/9 2024-04-24T01:53:02.318 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 8/9 2024-04-24T01:53:02.512 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mds-2:19.0.0-3227.g424d8c43.el9.x86_64 35/36 2024-04-24T01:53:02.534 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mds-2:19.0.0-3227.g424d8c43.el9.x86_64 35/36 2024-04-24T01:53:02.535 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T01:53:02.535 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-24T01:53:02.535 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-24T01:53:02.535 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-24T01:53:02.535 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:53:02.598 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-2:19.0.0-3227.g424d8c43.el9.x86_64 36/36 2024-04-24T01:53:02.687 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 9/9 2024-04-24T01:53:02.687 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:53:02.687 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-04-24T01:53:02.687 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-test-2:19.0.0-3227.g424d8c43.el9.x86_64 2024-04-24T01:53:02.687 INFO:teuthology.orchestra.run.smithi079.stdout: libasan8-13.2.1-6.3.el9.x86_64 2024-04-24T01:53:02.687 INFO:teuthology.orchestra.run.smithi079.stdout: libcephsqlite-2:19.0.0-3227.g424d8c43.el9.x86_64 2024-04-24T01:53:02.688 INFO:teuthology.orchestra.run.smithi079.stdout: libubsan-11.4.1-3.el9.x86_64 2024-04-24T01:53:02.688 INFO:teuthology.orchestra.run.smithi079.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2024-04-24T01:53:02.688 INFO:teuthology.orchestra.run.smithi079.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-24T01:53:02.688 INFO:teuthology.orchestra.run.smithi079.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-24T01:53:02.688 INFO:teuthology.orchestra.run.smithi079.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-24T01:53:02.688 INFO:teuthology.orchestra.run.smithi079.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2024-04-24T01:53:02.688 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:53:02.688 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-04-24T01:53:02.993 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph 2024-04-24T01:53:03.506 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:00:56 ago on Wed 24 Apr 2024 01:52:07 AM UTC. 2024-04-24T01:53:03.626 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-04-24T01:53:03.629 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-04-24T01:53:03.629 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2024-04-24T01:53:03.629 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-04-24T01:53:03.629 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-04-24T01:53:03.629 INFO:teuthology.orchestra.run.smithi079.stdout: ceph x86_64 2:19.0.0-3227.g424d8c43.el9 ceph 6.5 k 2024-04-24T01:53:03.629 INFO:teuthology.orchestra.run.smithi079.stdout:Installing dependencies: 2024-04-24T01:53:03.629 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mds x86_64 2:19.0.0-3227.g424d8c43.el9 ceph 2.2 M 2024-04-24T01:53:03.629 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr x86_64 2:19.0.0-3227.g424d8c43.el9 ceph 1.6 M 2024-04-24T01:53:03.630 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3227.g424d8c43.el9 ceph-noarch 249 k 2024-04-24T01:53:03.630 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mon x86_64 2:19.0.0-3227.g424d8c43.el9 ceph 5.0 M 2024-04-24T01:53:03.630 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-osd x86_64 2:19.0.0-3227.g424d8c43.el9 ceph 71 M 2024-04-24T01:53:03.630 INFO:teuthology.orchestra.run.smithi079.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-24T01:53:03.630 INFO:teuthology.orchestra.run.smithi079.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-24T01:53:03.630 INFO:teuthology.orchestra.run.smithi079.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-24T01:53:03.630 INFO:teuthology.orchestra.run.smithi079.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-24T01:53:03.630 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-24T01:53:03.630 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-24T01:53:03.630 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-24T01:53:03.630 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-24T01:53:03.630 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-24T01:53:03.630 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-24T01:53:03.631 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco-collections 2024-04-24T01:53:03.631 INFO:teuthology.orchestra.run.smithi079.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-04-24T01:53:03.631 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-24T01:53:03.631 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-24T01:53:03.631 INFO:teuthology.orchestra.run.smithi079.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-24T01:53:03.631 INFO:teuthology.orchestra.run.smithi079.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-24T01:53:03.631 INFO:teuthology.orchestra.run.smithi079.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-24T01:53:03.631 INFO:teuthology.orchestra.run.smithi079.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-24T01:53:03.631 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-24T01:53:03.631 INFO:teuthology.orchestra.run.smithi079.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-24T01:53:03.631 INFO:teuthology.orchestra.run.smithi079.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-24T01:53:03.631 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-24T01:53:03.631 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-24T01:53:03.631 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-24T01:53:03.631 INFO:teuthology.orchestra.run.smithi079.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-24T01:53:03.631 INFO:teuthology.orchestra.run.smithi079.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-24T01:53:03.632 INFO:teuthology.orchestra.run.smithi079.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-24T01:53:03.632 INFO:teuthology.orchestra.run.smithi079.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-24T01:53:03.632 INFO:teuthology.orchestra.run.smithi079.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-24T01:53:03.632 INFO:teuthology.orchestra.run.smithi079.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-24T01:53:03.632 INFO:teuthology.orchestra.run.smithi079.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-24T01:53:03.632 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:53:03.632 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-04-24T01:53:03.632 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-04-24T01:53:03.632 INFO:teuthology.orchestra.run.smithi079.stdout:Install 36 Packages 2024-04-24T01:53:03.632 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:53:03.633 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 84 M 2024-04-24T01:53:03.633 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 922 M 2024-04-24T01:53:03.633 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-04-24T01:53:04.866 INFO:teuthology.orchestra.run.smithi079.stdout:(1/36): ceph-19.0.0-3227.g424d8c43.el9.x86_64.r 32 kB/s | 6.5 kB 00:00 2024-04-24T01:53:05.283 INFO:teuthology.orchestra.run.smithi079.stdout:(2/36): ceph-mgr-19.0.0-3227.g424d8c43.el9.x86_ 2.5 MB/s | 1.6 MB 00:00 2024-04-24T01:53:05.367 INFO:teuthology.orchestra.run.smithi079.stdout:(3/36): ceph-mds-19.0.0-3227.g424d8c43.el9.x86_ 3.1 MB/s | 2.2 MB 00:00 2024-04-24T01:53:05.434 INFO:teuthology.orchestra.run.smithi079.stdout:(4/36): ceph-mgr-modules-core-19.0.0-3227.g424d 3.6 MB/s | 249 kB 00:00 2024-04-24T01:53:05.859 INFO:teuthology.orchestra.run.smithi079.stdout:(5/36): ceph-mon-19.0.0-3227.g424d8c43.el9.x86_ 5.1 MB/s | 5.0 MB 00:00 2024-04-24T01:53:06.359 INFO:teuthology.orchestra.run.smithi079.stdout:(6/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 274 kB/s | 253 kB 00:00 2024-04-24T01:53:07.109 INFO:teuthology.orchestra.run.smithi079.stdout:(7/36): python3-ply-3.11-14.el9.noarch.rpm 142 kB/s | 106 kB 00:00 2024-04-24T01:53:07.893 INFO:teuthology.orchestra.run.smithi079.stdout:(8/36): python3-pycparser-2.20-6.el9.noarch.rpm 173 kB/s | 135 kB 00:00 2024-04-24T01:53:08.043 INFO:teuthology.orchestra.run.smithi079.stdout:(9/36): python3-cryptography-36.0.1-4.el9.x86_6 575 kB/s | 1.2 MB 00:02 2024-04-24T01:53:09.060 INFO:teuthology.orchestra.run.smithi079.stdout:(10/36): ceph-osd-19.0.0-3227.g424d8c43.el9.x86 19 MB/s | 71 MB 00:03 2024-04-24T01:53:09.093 INFO:teuthology.orchestra.run.smithi079.stdout:(11/36): python3-requests-2.25.1-8.el9.noarch.r 104 kB/s | 125 kB 00:01 2024-04-24T01:53:09.396 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-2:19.0.0-3227.g424d8c43.el9.x86_64 36/36 2024-04-24T01:53:09.396 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-2:19.0.0-3227.g424d8c43.el9.x86_64 1/36 2024-04-24T01:53:09.396 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mds-2:19.0.0-3227.g424d8c43.el9.x86_64 2/36 2024-04-24T01:53:09.396 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mgr-2:19.0.0-3227.g424d8c43.el9.x86_64 3/36 2024-04-24T01:53:09.396 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mon-2:19.0.0-3227.g424d8c43.el9.x86_64 4/36 2024-04-24T01:53:09.396 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-osd-2:19.0.0-3227.g424d8c43.el9.x86_64 5/36 2024-04-24T01:53:09.396 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3227.g424d8c43.el9. 6/36 2024-04-24T01:53:09.396 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/36 2024-04-24T01:53:09.396 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/36 2024-04-24T01:53:09.396 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/36 2024-04-24T01:53:09.396 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/36 2024-04-24T01:53:09.396 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/36 2024-04-24T01:53:09.397 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 12/36 2024-04-24T01:53:09.397 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/36 2024-04-24T01:53:09.397 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/36 2024-04-24T01:53:09.397 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/36 2024-04-24T01:53:09.397 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/36 2024-04-24T01:53:09.397 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/36 2024-04-24T01:53:09.397 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-04-24T01:53:09.397 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-04-24T01:53:09.397 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-04-24T01:53:09.397 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-04-24T01:53:09.397 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-04-24T01:53:09.397 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-04-24T01:53:09.397 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-04-24T01:53:09.397 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-04-24T01:53:09.398 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-04-24T01:53:09.398 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-04-24T01:53:09.398 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-04-24T01:53:09.398 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-04-24T01:53:09.398 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-04-24T01:53:09.398 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-04-24T01:53:09.398 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-04-24T01:53:09.398 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-04-24T01:53:09.398 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-04-24T01:53:09.398 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-04-24T01:53:09.485 INFO:teuthology.orchestra.run.smithi079.stdout:(12/36): python3-urllib3-1.26.5-5.el9.noarch.rp 149 kB/s | 215 kB 00:01 2024-04-24T01:53:09.952 INFO:teuthology.orchestra.run.smithi079.stdout:(13/36): lua-5.4.4-4.el9.x86_64.rpm 211 kB/s | 188 kB 00:00 2024-04-24T01:53:09.977 INFO:teuthology.orchestra.run.smithi079.stdout:(14/36): python3-mako-1.1.4-6.el9.noarch.rpm 195 kB/s | 172 kB 00:00 2024-04-24T01:53:10.103 INFO:teuthology.orchestra.run.smithi079.stdout:(15/36): python3-markupsafe-1.1.1-12.el9.x86_64 56 kB/s | 35 kB 00:00 2024-04-24T01:53:10.186 INFO:teuthology.orchestra.run.smithi079.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 268 kB/s | 22 kB 00:00 2024-04-24T01:53:10.314 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-04-24T01:53:10.314 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:53:10.314 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-04-24T01:53:10.314 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-2:19.0.0-3227.g424d8c43.el9.x86_64 2024-04-24T01:53:10.315 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mds-2:19.0.0-3227.g424d8c43.el9.x86_64 2024-04-24T01:53:10.315 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-2:19.0.0-3227.g424d8c43.el9.x86_64 2024-04-24T01:53:10.315 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-modules-core-2:19.0.0-3227.g424d8c43.el9.noarch 2024-04-24T01:53:10.315 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mon-2:19.0.0-3227.g424d8c43.el9.x86_64 2024-04-24T01:53:10.315 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-osd-2:19.0.0-3227.g424d8c43.el9.x86_64 2024-04-24T01:53:10.315 INFO:teuthology.orchestra.run.smithi103.stdout: lua-5.4.4-4.el9.x86_64 2024-04-24T01:53:10.315 INFO:teuthology.orchestra.run.smithi103.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-24T01:53:10.315 INFO:teuthology.orchestra.run.smithi103.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-24T01:53:10.315 INFO:teuthology.orchestra.run.smithi103.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-24T01:53:10.315 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-24T01:53:10.315 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-24T01:53:10.315 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-24T01:53:10.315 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-24T01:53:10.315 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-24T01:53:10.315 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-24T01:53:10.316 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-24T01:53:10.316 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-24T01:53:10.316 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-24T01:53:10.316 INFO:teuthology.orchestra.run.smithi103.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-24T01:53:10.316 INFO:teuthology.orchestra.run.smithi103.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-24T01:53:10.316 INFO:teuthology.orchestra.run.smithi103.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-24T01:53:10.316 INFO:teuthology.orchestra.run.smithi103.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-24T01:53:10.316 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-24T01:53:10.316 INFO:teuthology.orchestra.run.smithi103.stdout: python3-ply-3.11-14.el9.noarch 2024-04-24T01:53:10.316 INFO:teuthology.orchestra.run.smithi103.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-24T01:53:10.316 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-24T01:53:10.316 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-24T01:53:10.316 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-24T01:53:10.316 INFO:teuthology.orchestra.run.smithi103.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-24T01:53:10.316 INFO:teuthology.orchestra.run.smithi103.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-24T01:53:10.316 INFO:teuthology.orchestra.run.smithi103.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-24T01:53:10.317 INFO:teuthology.orchestra.run.smithi103.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-24T01:53:10.317 INFO:teuthology.orchestra.run.smithi103.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-24T01:53:10.317 INFO:teuthology.orchestra.run.smithi103.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-24T01:53:10.317 INFO:teuthology.orchestra.run.smithi103.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-24T01:53:10.317 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:53:10.317 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-04-24T01:53:10.611 INFO:teuthology.orchestra.run.smithi079.stdout:(17/36): python3-toml-0.10.2-6.el9.noarch.rpm 66 kB/s | 42 kB 00:00 2024-04-24T01:53:10.637 INFO:teuthology.orchestra.run.smithi079.stdout:(18/36): python3-pytz-2021.1-5.el9.noarch.rpm 75 kB/s | 51 kB 00:00 2024-04-24T01:53:10.655 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-base 2024-04-24T01:53:10.737 INFO:teuthology.orchestra.run.smithi079.stdout:(19/36): luarocks-3.9.2-1.el9.noarch.rpm 303 kB/s | 151 kB 00:00 2024-04-24T01:53:10.895 INFO:teuthology.orchestra.run.smithi079.stdout:(20/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 153 kB/s | 43 kB 00:00 2024-04-24T01:53:10.929 INFO:teuthology.orchestra.run.smithi079.stdout:(21/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.8 MB/s | 358 kB 00:00 2024-04-24T01:53:10.996 INFO:teuthology.orchestra.run.smithi079.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 265 kB/s | 18 kB 00:00 2024-04-24T01:53:11.021 INFO:teuthology.orchestra.run.smithi079.stdout:(23/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 85 kB/s | 11 kB 00:00 2024-04-24T01:53:11.046 INFO:teuthology.orchestra.run.smithi079.stdout:(24/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 419 kB/s | 172 kB 00:00 2024-04-24T01:53:11.105 INFO:teuthology.orchestra.run.smithi079.stdout:(25/36): python3-jaraco-collections-3.0.0-8.el9 213 kB/s | 23 kB 00:00 2024-04-24T01:53:11.130 INFO:teuthology.orchestra.run.smithi079.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 179 kB/s | 19 kB 00:00 2024-04-24T01:53:11.156 INFO:teuthology.orchestra.run.smithi079.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 180 kB/s | 20 kB 00:00 2024-04-24T01:53:11.167 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:25 ago on Wed 24 Apr 2024 01:51:46 AM UTC. 2024-04-24T01:53:11.239 INFO:teuthology.orchestra.run.smithi079.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 346 kB/s | 46 kB 00:00 2024-04-24T01:53:11.245 INFO:teuthology.orchestra.run.smithi103.stdout:Package ceph-base-2:19.0.0-3227.g424d8c43.el9.x86_64 is already installed. 2024-04-24T01:53:11.287 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-04-24T01:53:11.288 INFO:teuthology.orchestra.run.smithi103.stdout:Nothing to do. 2024-04-24T01:53:11.288 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-04-24T01:53:11.314 INFO:teuthology.orchestra.run.smithi079.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 429 kB/s | 79 kB 00:00 2024-04-24T01:53:11.340 INFO:teuthology.orchestra.run.smithi079.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 164 kB/s | 16 kB 00:00 2024-04-24T01:53:11.364 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install cephadm 2024-04-24T01:53:11.365 INFO:teuthology.orchestra.run.smithi079.stdout:(31/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.3 MB/s | 272 kB 00:00 2024-04-24T01:53:11.449 INFO:teuthology.orchestra.run.smithi079.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 330 kB/s | 36 kB 00:00 2024-04-24T01:53:11.474 INFO:teuthology.orchestra.run.smithi079.stdout:(33/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 565 kB/s | 90 kB 00:00 2024-04-24T01:53:11.499 INFO:teuthology.orchestra.run.smithi079.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.7 MB/s | 230 kB 00:00 2024-04-24T01:53:11.591 INFO:teuthology.orchestra.run.smithi079.stdout:(35/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.9 MB/s | 427 kB 00:00 2024-04-24T01:53:11.608 INFO:teuthology.orchestra.run.smithi079.stdout:(36/36): python3-zc-lockfile-2.0-10.el9.noarch. 149 kB/s | 20 kB 00:00 2024-04-24T01:53:11.614 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-04-24T01:53:11.614 INFO:teuthology.orchestra.run.smithi079.stdout:Total 11 MB/s | 84 MB 00:07 2024-04-24T01:53:11.750 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-04-24T01:53:11.791 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-04-24T01:53:11.791 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-04-24T01:53:11.885 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:25 ago on Wed 24 Apr 2024 01:51:46 AM UTC. 2024-04-24T01:53:12.009 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-04-24T01:53:12.010 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-24T01:53:12.011 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repository Size 2024-04-24T01:53:12.011 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-24T01:53:12.011 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-04-24T01:53:12.011 INFO:teuthology.orchestra.run.smithi103.stdout: cephadm noarch 2:19.0.0-3227.g424d8c43.el9 ceph-noarch 762 k 2024-04-24T01:53:12.011 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:53:12.011 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-04-24T01:53:12.011 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-24T01:53:12.011 INFO:teuthology.orchestra.run.smithi103.stdout:Install 1 Package 2024-04-24T01:53:12.011 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:53:12.011 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 762 k 2024-04-24T01:53:12.011 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 769 k 2024-04-24T01:53:12.011 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-04-24T01:53:12.403 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-04-24T01:53:12.404 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-04-24T01:53:12.437 INFO:teuthology.orchestra.run.smithi103.stdout:cephadm-19.0.0-3227.g424d8c43.el9.noarch.rpm 1.8 MB/s | 762 kB 00:00 2024-04-24T01:53:12.437 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-04-24T01:53:12.438 INFO:teuthology.orchestra.run.smithi103.stdout:Total 1.7 MB/s | 762 kB 00:00 2024-04-24T01:53:12.438 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-04-24T01:53:12.444 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-04-24T01:53:12.444 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-04-24T01:53:12.450 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-04-24T01:53:12.450 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-04-24T01:53:12.521 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-04-24T01:53:13.021 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: cephadm-2:19.0.0-3227.g424d8c43.el9.noarch 1/1 2024-04-24T01:53:13.098 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : cephadm-2:19.0.0-3227.g424d8c43.el9.noarch 1/1 2024-04-24T01:53:13.418 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: cephadm-2:19.0.0-3227.g424d8c43.el9.noarch 1/1 2024-04-24T01:53:13.641 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : cephadm-2:19.0.0-3227.g424d8c43.el9.noarch 1/1 2024-04-24T01:53:13.641 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:53:13.641 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-04-24T01:53:13.641 INFO:teuthology.orchestra.run.smithi103.stdout: cephadm-2:19.0.0-3227.g424d8c43.el9.noarch 2024-04-24T01:53:13.642 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:53:13.642 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-04-24T01:53:13.849 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-immutable-object-cache 2024-04-24T01:53:13.965 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-04-24T01:53:14.027 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-04-24T01:53:14.059 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-04-24T01:53:14.096 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-04-24T01:53:14.167 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-04-24T01:53:14.211 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-04-24T01:53:14.244 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-04-24T01:53:14.344 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-04-24T01:53:14.373 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:28 ago on Wed 24 Apr 2024 01:51:46 AM UTC. 2024-04-24T01:53:14.422 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-04-24T01:53:14.463 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-04-24T01:53:14.496 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-04-24T01:53:14.496 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-24T01:53:14.496 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repo Size 2024-04-24T01:53:14.496 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-24T01:53:14.496 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-04-24T01:53:14.497 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3227.g424d8c43.el9 ceph 157 k 2024-04-24T01:53:14.497 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:53:14.497 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-04-24T01:53:14.497 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-24T01:53:14.497 INFO:teuthology.orchestra.run.smithi103.stdout:Install 1 Package 2024-04-24T01:53:14.497 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:53:14.497 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 157 k 2024-04-24T01:53:14.497 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 640 k 2024-04-24T01:53:14.498 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-04-24T01:53:14.507 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-04-24T01:53:14.537 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/36 2024-04-24T01:53:14.571 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/36 2024-04-24T01:53:14.605 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/36 2024-04-24T01:53:14.645 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/36 2024-04-24T01:53:14.680 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/36 2024-04-24T01:53:14.715 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/36 2024-04-24T01:53:14.752 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/36 2024-04-24T01:53:14.825 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 18/36 2024-04-24T01:53:14.870 INFO:teuthology.orchestra.run.smithi103.stdout:ceph-immutable-object-cache-19.0.0-3227.g424d8c 422 kB/s | 157 kB 00:00 2024-04-24T01:53:14.870 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-04-24T01:53:14.871 INFO:teuthology.orchestra.run.smithi103.stdout:Total 420 kB/s | 157 kB 00:00 2024-04-24T01:53:14.871 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-04-24T01:53:14.881 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-04-24T01:53:14.881 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-04-24T01:53:14.938 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-04-24T01:53:14.938 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-04-24T01:53:14.963 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 19/36 2024-04-24T01:53:15.043 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 20/36 2024-04-24T01:53:15.092 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-04-24T01:53:15.096 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 21/36 2024-04-24T01:53:15.135 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 22/36 2024-04-24T01:53:15.162 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3227.g424d8c43. 1/1 2024-04-24T01:53:15.184 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-ply-3.11-14.el9.noarch 23/36 2024-04-24T01:53:15.187 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3227.g424d8c43. 1/1 2024-04-24T01:53:15.187 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T01:53:15.187 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-24T01:53:15.187 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:53:15.239 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 24/36 2024-04-24T01:53:15.425 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 25/36 2024-04-24T01:53:15.473 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 26/36 2024-04-24T01:53:15.542 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 27/36 2024-04-24T01:53:15.630 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 28/36 2024-04-24T01:53:15.672 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 29/36 2024-04-24T01:53:15.788 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 30/36 2024-04-24T01:53:15.981 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3227.g424d8c43.el9. 31/36 2024-04-24T01:53:16.007 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mgr-2:19.0.0-3227.g424d8c43.el9.x86_64 32/36 2024-04-24T01:53:16.032 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3227.g424d8c43.el9.x86_64 32/36 2024-04-24T01:53:16.033 INFO:teuthology.orchestra.run.smithi079.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T01:53:16.033 INFO:teuthology.orchestra.run.smithi079.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-24T01:53:16.033 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-24T01:53:16.033 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-24T01:53:16.033 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:53:16.052 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3227.g424d8c43. 1/1 2024-04-24T01:53:16.052 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:53:16.052 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-04-24T01:53:16.052 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-immutable-object-cache-2:19.0.0-3227.g424d8c43.el9.x86_64 2024-04-24T01:53:16.052 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:53:16.052 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-04-24T01:53:16.281 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-mgr 2024-04-24T01:53:16.798 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:30 ago on Wed 24 Apr 2024 01:51:46 AM UTC. 2024-04-24T01:53:16.878 INFO:teuthology.orchestra.run.smithi103.stdout:Package ceph-mgr-2:19.0.0-3227.g424d8c43.el9.x86_64 is already installed. 2024-04-24T01:53:16.922 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-04-24T01:53:16.923 INFO:teuthology.orchestra.run.smithi103.stdout:Nothing to do. 2024-04-24T01:53:16.924 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-04-24T01:53:16.989 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-mgr-dashboard 2024-04-24T01:53:17.499 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:31 ago on Wed 24 Apr 2024 01:51:46 AM UTC. 2024-04-24T01:53:17.619 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-04-24T01:53:17.620 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-24T01:53:17.620 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repository Size 2024-04-24T01:53:17.620 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-24T01:53:17.620 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-04-24T01:53:17.620 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3227.g424d8c43.el9 ceph-noarch 3.5 M 2024-04-24T01:53:17.620 INFO:teuthology.orchestra.run.smithi103.stdout:Installing dependencies: 2024-04-24T01:53:17.620 INFO:teuthology.orchestra.run.smithi103.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-24T01:53:17.620 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3227.g424d8c43.el9 ceph-noarch 34 k 2024-04-24T01:53:17.620 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3227.g424d8c43.el9 ceph-noarch 16 k 2024-04-24T01:53:17.620 INFO:teuthology.orchestra.run.smithi103.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-24T01:53:17.620 INFO:teuthology.orchestra.run.smithi103.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-24T01:53:17.621 INFO:teuthology.orchestra.run.smithi103.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-24T01:53:17.621 INFO:teuthology.orchestra.run.smithi103.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-24T01:53:17.621 INFO:teuthology.orchestra.run.smithi103.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-24T01:53:17.621 INFO:teuthology.orchestra.run.smithi103.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-24T01:53:17.621 INFO:teuthology.orchestra.run.smithi103.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-24T01:53:17.621 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:53:17.621 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-04-24T01:53:17.621 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-24T01:53:17.621 INFO:teuthology.orchestra.run.smithi103.stdout:Install 11 Packages 2024-04-24T01:53:17.621 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:53:17.622 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 7.6 M 2024-04-24T01:53:17.622 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 100 M 2024-04-24T01:53:17.622 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-04-24T01:53:18.166 INFO:teuthology.orchestra.run.smithi103.stdout:(1/11): ceph-prometheus-alerts-19.0.0-3227.g424 66 kB/s | 16 kB 00:00 2024-04-24T01:53:18.191 INFO:teuthology.orchestra.run.smithi103.stdout:(2/11): ceph-grafana-dashboards-19.0.0-3227.g42 125 kB/s | 34 kB 00:00 2024-04-24T01:53:18.350 INFO:teuthology.orchestra.run.smithi103.stdout:(3/11): python3-protobuf-3.14.0-13.el9.noarch.r 1.4 MB/s | 269 kB 00:00 2024-04-24T01:53:18.500 INFO:teuthology.orchestra.run.smithi103.stdout:(4/11): protobuf-compiler-3.14.0-13.el9.x86_64. 2.7 MB/s | 863 kB 00:00 2024-04-24T01:53:18.609 INFO:teuthology.orchestra.run.smithi103.stdout:(5/11): ceph-mgr-dashboard-19.0.0-3227.g424d8c4 5.1 MB/s | 3.5 MB 00:00 2024-04-24T01:53:18.818 INFO:teuthology.orchestra.run.smithi103.stdout:(6/11): grpc-data-1.46.7-10.el9.noarch.rpm 61 kB/s | 19 kB 00:00 2024-04-24T01:53:19.368 INFO:teuthology.orchestra.run.smithi103.stdout:(7/11): python3-grpcio-tools-1.46.7-10.el9.x86_ 263 kB/s | 144 kB 00:00 2024-04-24T01:53:19.493 INFO:teuthology.orchestra.run.smithi103.stdout:(8/11): python3-repoze-lru-0.7-16.el9.noarch.rp 246 kB/s | 31 kB 00:00 2024-04-24T01:53:19.669 INFO:teuthology.orchestra.run.smithi103.stdout:(9/11): abseil-cpp-20211102.0-3.el9.x86_64.rpm 416 kB/s | 548 kB 00:01 2024-04-24T01:53:19.877 INFO:teuthology.orchestra.run.smithi103.stdout:(10/11): python3-routes-2.5.1-5.el9.noarch.rpm 491 kB/s | 188 kB 00:00 2024-04-24T01:53:22.077 INFO:teuthology.orchestra.run.smithi103.stdout:(11/11): python3-grpcio-1.46.7-10.el9.x86_64.rp 602 kB/s | 2.0 MB 00:03 2024-04-24T01:53:22.080 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-04-24T01:53:22.080 INFO:teuthology.orchestra.run.smithi103.stdout:Total 1.7 MB/s | 7.6 MB 00:04 2024-04-24T01:53:22.113 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-04-24T01:53:22.132 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-04-24T01:53:22.132 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-04-24T01:53:22.280 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-04-24T01:53:22.281 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-04-24T01:53:22.780 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-04-24T01:53:22.858 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/11 2024-04-24T01:53:22.889 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/11 2024-04-24T01:53:22.968 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 3/11 2024-04-24T01:53:23.109 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 4/11 2024-04-24T01:53:23.159 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 5/11 2024-04-24T01:53:23.244 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 6/11 2024-04-24T01:53:23.286 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 7/11 2024-04-24T01:53:23.314 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 8/11 2024-04-24T01:53:23.347 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3227.g424d8c43.el9 9/11 2024-04-24T01:53:24.395 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3227.g424d8c43.el 10/11 2024-04-24T01:53:24.439 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3227.g424d8c43.el9.noa 11/11 2024-04-24T01:53:25.739 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3227.g424d8c43.el9.noa 11/11 2024-04-24T01:53:25.739 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3227.g424d8c43.el 1/11 2024-04-24T01:53:25.739 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3227.g424d8c43.el9.noa 2/11 2024-04-24T01:53:25.739 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3227.g424d8c43.el9 3/11 2024-04-24T01:53:25.739 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 4/11 2024-04-24T01:53:25.739 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 5/11 2024-04-24T01:53:25.739 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 6/11 2024-04-24T01:53:25.739 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 7/11 2024-04-24T01:53:25.739 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 8/11 2024-04-24T01:53:25.739 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/11 2024-04-24T01:53:25.740 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 10/11 2024-04-24T01:53:26.024 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 11/11 2024-04-24T01:53:26.024 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:53:26.024 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-04-24T01:53:26.024 INFO:teuthology.orchestra.run.smithi103.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-24T01:53:26.024 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-grafana-dashboards-2:19.0.0-3227.g424d8c43.el9.noarch 2024-04-24T01:53:26.024 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-dashboard-2:19.0.0-3227.g424d8c43.el9.noarch 2024-04-24T01:53:26.025 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-prometheus-alerts-2:19.0.0-3227.g424d8c43.el9.noarch 2024-04-24T01:53:26.025 INFO:teuthology.orchestra.run.smithi103.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-24T01:53:26.025 INFO:teuthology.orchestra.run.smithi103.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-24T01:53:26.025 INFO:teuthology.orchestra.run.smithi103.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-24T01:53:26.025 INFO:teuthology.orchestra.run.smithi103.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-24T01:53:26.025 INFO:teuthology.orchestra.run.smithi103.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-24T01:53:26.025 INFO:teuthology.orchestra.run.smithi103.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-24T01:53:26.025 INFO:teuthology.orchestra.run.smithi103.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-24T01:53:26.025 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:53:26.025 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-04-24T01:53:26.272 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-24T01:53:26.775 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:40 ago on Wed 24 Apr 2024 01:51:46 AM UTC. 2024-04-24T01:53:26.848 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-osd-2:19.0.0-3227.g424d8c43.el9.x86_64 33/36 2024-04-24T01:53:26.874 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-osd-2:19.0.0-3227.g424d8c43.el9.x86_64 33/36 2024-04-24T01:53:26.874 INFO:teuthology.orchestra.run.smithi079.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T01:53:26.874 INFO:teuthology.orchestra.run.smithi079.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-24T01:53:26.874 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-24T01:53:26.874 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-24T01:53:26.874 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:53:26.896 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-04-24T01:53:26.897 INFO:teuthology.orchestra.run.smithi103.stdout:======================================================================================= 2024-04-24T01:53:26.897 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repository Size 2024-04-24T01:53:26.897 INFO:teuthology.orchestra.run.smithi103.stdout:======================================================================================= 2024-04-24T01:53:26.897 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-04-24T01:53:26.897 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3227.g424d8c43.el9 ceph-noarch 7.4 M 2024-04-24T01:53:26.898 INFO:teuthology.orchestra.run.smithi103.stdout:Installing dependencies: 2024-04-24T01:53:26.898 INFO:teuthology.orchestra.run.smithi103.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-24T01:53:26.898 INFO:teuthology.orchestra.run.smithi103.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-24T01:53:26.898 INFO:teuthology.orchestra.run.smithi103.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-24T01:53:26.898 INFO:teuthology.orchestra.run.smithi103.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-24T01:53:26.898 INFO:teuthology.orchestra.run.smithi103.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-24T01:53:26.898 INFO:teuthology.orchestra.run.smithi103.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-24T01:53:26.898 INFO:teuthology.orchestra.run.smithi103.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-24T01:53:26.898 INFO:teuthology.orchestra.run.smithi103.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-24T01:53:26.898 INFO:teuthology.orchestra.run.smithi103.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-24T01:53:26.898 INFO:teuthology.orchestra.run.smithi103.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-24T01:53:26.898 INFO:teuthology.orchestra.run.smithi103.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-24T01:53:26.898 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:53:26.898 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-04-24T01:53:26.899 INFO:teuthology.orchestra.run.smithi103.stdout:======================================================================================= 2024-04-24T01:53:26.899 INFO:teuthology.orchestra.run.smithi103.stdout:Install 12 Packages 2024-04-24T01:53:26.899 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:53:26.899 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 38 M 2024-04-24T01:53:26.899 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 209 M 2024-04-24T01:53:26.899 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-04-24T01:53:27.360 INFO:teuthology.orchestra.run.smithi103.stdout:(1/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.2 MB/s | 195 kB 00:00 2024-04-24T01:53:27.425 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mon-2:19.0.0-3227.g424d8c43.el9.x86_64 34/36 2024-04-24T01:53:27.435 INFO:teuthology.orchestra.run.smithi103.stdout:(2/12): libgfortran-11.4.1-3.el9.x86_64.rpm 3.4 MB/s | 803 kB 00:00 2024-04-24T01:53:27.451 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-mon-2:19.0.0-3227.g424d8c43.el9.x86_64 34/36 2024-04-24T01:53:27.451 INFO:teuthology.orchestra.run.smithi079.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T01:53:27.451 INFO:teuthology.orchestra.run.smithi079.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-24T01:53:27.451 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-24T01:53:27.451 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-24T01:53:27.451 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:53:27.700 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mds-2:19.0.0-3227.g424d8c43.el9.x86_64 35/36 2024-04-24T01:53:27.724 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-mds-2:19.0.0-3227.g424d8c43.el9.x86_64 35/36 2024-04-24T01:53:27.724 INFO:teuthology.orchestra.run.smithi079.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T01:53:27.724 INFO:teuthology.orchestra.run.smithi079.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-24T01:53:27.724 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-24T01:53:27.724 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-24T01:53:27.725 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:53:27.797 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-2:19.0.0-3227.g424d8c43.el9.x86_64 36/36 2024-04-24T01:53:28.027 INFO:teuthology.orchestra.run.smithi103.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 49 kB/s | 33 kB 00:00 2024-04-24T01:53:28.603 INFO:teuthology.orchestra.run.smithi103.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 32 kB/s | 18 kB 00:00 2024-04-24T01:53:29.236 INFO:teuthology.orchestra.run.smithi103.stdout:(5/12): ceph-mgr-diskprediction-local-19.0.0-32 3.7 MB/s | 7.4 MB 00:02 2024-04-24T01:53:29.319 INFO:teuthology.orchestra.run.smithi103.stdout:(6/12): openblas-0.3.21-2.el9.x86_64.rpm 49 kB/s | 35 kB 00:00 2024-04-24T01:53:30.387 INFO:teuthology.orchestra.run.smithi103.stdout:(7/12): python3-devel-3.9.18-3.el9.x86_64.rpm 229 kB/s | 244 kB 00:01 2024-04-24T01:53:32.345 INFO:teuthology.orchestra.run.smithi103.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 623 kB/s | 3.0 MB 00:04 2024-04-24T01:53:33.237 INFO:teuthology.orchestra.run.smithi103.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 281 kB/s | 250 kB 00:00 2024-04-24T01:53:34.688 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-2:19.0.0-3227.g424d8c43.el9.x86_64 36/36 2024-04-24T01:53:34.689 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-2:19.0.0-3227.g424d8c43.el9.x86_64 1/36 2024-04-24T01:53:34.689 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mds-2:19.0.0-3227.g424d8c43.el9.x86_64 2/36 2024-04-24T01:53:34.689 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mgr-2:19.0.0-3227.g424d8c43.el9.x86_64 3/36 2024-04-24T01:53:34.689 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mon-2:19.0.0-3227.g424d8c43.el9.x86_64 4/36 2024-04-24T01:53:34.689 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-osd-2:19.0.0-3227.g424d8c43.el9.x86_64 5/36 2024-04-24T01:53:34.689 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3227.g424d8c43.el9. 6/36 2024-04-24T01:53:34.689 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/36 2024-04-24T01:53:34.689 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/36 2024-04-24T01:53:34.689 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/36 2024-04-24T01:53:34.689 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/36 2024-04-24T01:53:34.689 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/36 2024-04-24T01:53:34.690 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 12/36 2024-04-24T01:53:34.690 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/36 2024-04-24T01:53:34.690 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/36 2024-04-24T01:53:34.690 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/36 2024-04-24T01:53:34.690 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/36 2024-04-24T01:53:34.690 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/36 2024-04-24T01:53:34.690 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-04-24T01:53:34.690 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-04-24T01:53:34.690 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-04-24T01:53:34.690 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-04-24T01:53:34.690 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-04-24T01:53:34.690 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-04-24T01:53:34.690 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-04-24T01:53:34.690 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-04-24T01:53:34.691 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-04-24T01:53:34.691 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-04-24T01:53:34.691 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-04-24T01:53:34.691 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-04-24T01:53:34.691 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-04-24T01:53:34.692 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-04-24T01:53:34.693 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-04-24T01:53:34.693 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-04-24T01:53:34.693 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-04-24T01:53:34.693 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-04-24T01:53:35.626 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-04-24T01:53:35.626 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:53:35.626 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-04-24T01:53:35.626 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-2:19.0.0-3227.g424d8c43.el9.x86_64 2024-04-24T01:53:35.627 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mds-2:19.0.0-3227.g424d8c43.el9.x86_64 2024-04-24T01:53:35.627 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-2:19.0.0-3227.g424d8c43.el9.x86_64 2024-04-24T01:53:35.627 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-modules-core-2:19.0.0-3227.g424d8c43.el9.noarch 2024-04-24T01:53:35.627 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mon-2:19.0.0-3227.g424d8c43.el9.x86_64 2024-04-24T01:53:35.627 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-osd-2:19.0.0-3227.g424d8c43.el9.x86_64 2024-04-24T01:53:35.627 INFO:teuthology.orchestra.run.smithi079.stdout: lua-5.4.4-4.el9.x86_64 2024-04-24T01:53:35.627 INFO:teuthology.orchestra.run.smithi079.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-24T01:53:35.627 INFO:teuthology.orchestra.run.smithi079.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-24T01:53:35.627 INFO:teuthology.orchestra.run.smithi079.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-24T01:53:35.627 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-24T01:53:35.627 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-24T01:53:35.627 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-24T01:53:35.628 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-24T01:53:35.628 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-24T01:53:35.628 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-24T01:53:35.628 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-24T01:53:35.628 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-24T01:53:35.628 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-24T01:53:35.628 INFO:teuthology.orchestra.run.smithi079.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-24T01:53:35.628 INFO:teuthology.orchestra.run.smithi079.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-24T01:53:35.628 INFO:teuthology.orchestra.run.smithi079.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-24T01:53:35.628 INFO:teuthology.orchestra.run.smithi079.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-24T01:53:35.628 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-24T01:53:35.629 INFO:teuthology.orchestra.run.smithi079.stdout: python3-ply-3.11-14.el9.noarch 2024-04-24T01:53:35.629 INFO:teuthology.orchestra.run.smithi079.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-24T01:53:35.629 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-24T01:53:35.629 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-24T01:53:35.629 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-24T01:53:35.629 INFO:teuthology.orchestra.run.smithi079.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-24T01:53:35.629 INFO:teuthology.orchestra.run.smithi079.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-24T01:53:35.629 INFO:teuthology.orchestra.run.smithi079.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-24T01:53:35.629 INFO:teuthology.orchestra.run.smithi079.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-24T01:53:35.629 INFO:teuthology.orchestra.run.smithi079.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-24T01:53:35.629 INFO:teuthology.orchestra.run.smithi079.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-24T01:53:35.629 INFO:teuthology.orchestra.run.smithi079.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-24T01:53:35.629 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:53:35.630 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-04-24T01:53:35.978 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-base 2024-04-24T01:53:36.078 INFO:teuthology.orchestra.run.smithi103.stdout:(10/12): openblas-openmp-0.3.21-2.el9.x86_64.rp 709 kB/s | 4.7 MB 00:06 2024-04-24T01:53:36.489 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:29 ago on Wed 24 Apr 2024 01:52:07 AM UTC. 2024-04-24T01:53:36.569 INFO:teuthology.orchestra.run.smithi079.stdout:Package ceph-base-2:19.0.0-3227.g424d8c43.el9.x86_64 is already installed. 2024-04-24T01:53:36.610 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-04-24T01:53:36.612 INFO:teuthology.orchestra.run.smithi079.stdout:Nothing to do. 2024-04-24T01:53:36.612 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-04-24T01:53:36.695 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install cephadm 2024-04-24T01:53:37.199 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:30 ago on Wed 24 Apr 2024 01:52:07 AM UTC. 2024-04-24T01:53:37.318 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-04-24T01:53:37.319 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-04-24T01:53:37.319 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2024-04-24T01:53:37.319 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-04-24T01:53:37.320 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-04-24T01:53:37.320 INFO:teuthology.orchestra.run.smithi079.stdout: cephadm noarch 2:19.0.0-3227.g424d8c43.el9 ceph-noarch 762 k 2024-04-24T01:53:37.320 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:53:37.320 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-04-24T01:53:37.320 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-04-24T01:53:37.320 INFO:teuthology.orchestra.run.smithi079.stdout:Install 1 Package 2024-04-24T01:53:37.320 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:53:37.320 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 762 k 2024-04-24T01:53:37.320 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 769 k 2024-04-24T01:53:37.320 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-04-24T01:53:37.623 INFO:teuthology.orchestra.run.smithi079.stdout:cephadm-19.0.0-3227.g424d8c43.el9.noarch.rpm 2.5 MB/s | 762 kB 00:00 2024-04-24T01:53:37.623 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-04-24T01:53:37.624 INFO:teuthology.orchestra.run.smithi079.stdout:Total 2.5 MB/s | 762 kB 00:00 2024-04-24T01:53:37.624 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-04-24T01:53:37.629 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-04-24T01:53:37.629 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-04-24T01:53:37.635 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-04-24T01:53:37.635 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-04-24T01:53:37.708 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-04-24T01:53:37.812 INFO:teuthology.orchestra.run.smithi103.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 703 kB/s | 5.1 MB 00:07 2024-04-24T01:53:38.148 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: cephadm-2:19.0.0-3227.g424d8c43.el9.noarch 1/1 2024-04-24T01:53:38.243 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : cephadm-2:19.0.0-3227.g424d8c43.el9.noarch 1/1 2024-04-24T01:53:38.605 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: cephadm-2:19.0.0-3227.g424d8c43.el9.noarch 1/1 2024-04-24T01:53:38.837 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : cephadm-2:19.0.0-3227.g424d8c43.el9.noarch 1/1 2024-04-24T01:53:38.837 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:53:38.837 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-04-24T01:53:38.837 INFO:teuthology.orchestra.run.smithi079.stdout: cephadm-2:19.0.0-3227.g424d8c43.el9.noarch 2024-04-24T01:53:38.838 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:53:38.838 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-04-24T01:53:39.085 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-immutable-object-cache 2024-04-24T01:53:39.599 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:32 ago on Wed 24 Apr 2024 01:52:07 AM UTC. 2024-04-24T01:53:39.720 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-04-24T01:53:39.721 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-04-24T01:53:39.721 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repo Size 2024-04-24T01:53:39.721 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-04-24T01:53:39.721 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-04-24T01:53:39.721 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3227.g424d8c43.el9 ceph 157 k 2024-04-24T01:53:39.721 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:53:39.721 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-04-24T01:53:39.721 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-04-24T01:53:39.721 INFO:teuthology.orchestra.run.smithi079.stdout:Install 1 Package 2024-04-24T01:53:39.721 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:53:39.721 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 157 k 2024-04-24T01:53:39.721 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 640 k 2024-04-24T01:53:39.722 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-04-24T01:53:39.929 INFO:teuthology.orchestra.run.smithi079.stdout:ceph-immutable-object-cache-19.0.0-3227.g424d8c 757 kB/s | 157 kB 00:00 2024-04-24T01:53:39.930 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-04-24T01:53:39.930 INFO:teuthology.orchestra.run.smithi079.stdout:Total 751 kB/s | 157 kB 00:00 2024-04-24T01:53:39.930 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-04-24T01:53:39.940 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-04-24T01:53:39.940 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-04-24T01:53:39.997 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-04-24T01:53:39.997 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-04-24T01:53:40.145 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-04-24T01:53:40.205 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3227.g424d8c43. 1/1 2024-04-24T01:53:40.230 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3227.g424d8c43. 1/1 2024-04-24T01:53:40.230 INFO:teuthology.orchestra.run.smithi079.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T01:53:40.230 INFO:teuthology.orchestra.run.smithi079.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-24T01:53:40.230 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:53:41.114 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3227.g424d8c43. 1/1 2024-04-24T01:53:41.114 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:53:41.114 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-04-24T01:53:41.114 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-immutable-object-cache-2:19.0.0-3227.g424d8c43.el9.x86_64 2024-04-24T01:53:41.114 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:53:41.115 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-04-24T01:53:41.339 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-mgr 2024-04-24T01:53:41.861 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:34 ago on Wed 24 Apr 2024 01:52:07 AM UTC. 2024-04-24T01:53:41.942 INFO:teuthology.orchestra.run.smithi079.stdout:Package ceph-mgr-2:19.0.0-3227.g424d8c43.el9.x86_64 is already installed. 2024-04-24T01:53:41.984 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-04-24T01:53:41.984 INFO:teuthology.orchestra.run.smithi079.stdout:Nothing to do. 2024-04-24T01:53:41.985 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-04-24T01:53:42.068 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-mgr-dashboard 2024-04-24T01:53:42.573 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:35 ago on Wed 24 Apr 2024 01:52:07 AM UTC. 2024-04-24T01:53:42.693 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-04-24T01:53:42.694 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-04-24T01:53:42.695 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2024-04-24T01:53:42.695 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-04-24T01:53:42.695 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-04-24T01:53:42.695 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3227.g424d8c43.el9 ceph-noarch 3.5 M 2024-04-24T01:53:42.695 INFO:teuthology.orchestra.run.smithi079.stdout:Installing dependencies: 2024-04-24T01:53:42.695 INFO:teuthology.orchestra.run.smithi079.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-24T01:53:42.695 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3227.g424d8c43.el9 ceph-noarch 34 k 2024-04-24T01:53:42.695 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3227.g424d8c43.el9 ceph-noarch 16 k 2024-04-24T01:53:42.695 INFO:teuthology.orchestra.run.smithi079.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-24T01:53:42.695 INFO:teuthology.orchestra.run.smithi079.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-24T01:53:42.695 INFO:teuthology.orchestra.run.smithi079.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-24T01:53:42.695 INFO:teuthology.orchestra.run.smithi079.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-24T01:53:42.695 INFO:teuthology.orchestra.run.smithi079.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-24T01:53:42.696 INFO:teuthology.orchestra.run.smithi079.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-24T01:53:42.696 INFO:teuthology.orchestra.run.smithi079.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-24T01:53:42.696 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:53:42.696 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-04-24T01:53:42.696 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-04-24T01:53:42.696 INFO:teuthology.orchestra.run.smithi079.stdout:Install 11 Packages 2024-04-24T01:53:42.696 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:53:42.696 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 7.6 M 2024-04-24T01:53:42.697 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 100 M 2024-04-24T01:53:42.697 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-04-24T01:53:43.166 INFO:teuthology.orchestra.run.smithi079.stdout:(1/11): ceph-prometheus-alerts-19.0.0-3227.g424 64 kB/s | 16 kB 00:00 2024-04-24T01:53:43.182 INFO:teuthology.orchestra.run.smithi079.stdout:(2/11): ceph-grafana-dashboards-19.0.0-3227.g42 126 kB/s | 34 kB 00:00 2024-04-24T01:53:43.341 INFO:teuthology.orchestra.run.smithi079.stdout:(3/11): python3-protobuf-3.14.0-13.el9.noarch.r 1.5 MB/s | 269 kB 00:00 2024-04-24T01:53:43.458 INFO:teuthology.orchestra.run.smithi079.stdout:(4/11): abseil-cpp-20211102.0-3.el9.x86_64.rpm 4.6 MB/s | 548 kB 00:00 2024-04-24T01:53:43.517 INFO:teuthology.orchestra.run.smithi079.stdout:(5/11): ceph-mgr-dashboard-19.0.0-3227.g424d8c4 5.8 MB/s | 3.5 MB 00:00 2024-04-24T01:53:43.542 INFO:teuthology.orchestra.run.smithi079.stdout:(6/11): grpc-data-1.46.7-10.el9.noarch.rpm 232 kB/s | 19 kB 00:00 2024-04-24T01:53:43.625 INFO:teuthology.orchestra.run.smithi079.stdout:(7/11): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.7 MB/s | 144 kB 00:00 2024-04-24T01:53:43.667 INFO:teuthology.orchestra.run.smithi079.stdout:(8/11): python3-repoze-lru-0.7-16.el9.noarch.rp 740 kB/s | 31 kB 00:00 2024-04-24T01:53:43.726 INFO:teuthology.orchestra.run.smithi079.stdout:(9/11): python3-grpcio-1.46.7-10.el9.x86_64.rpm 9.7 MB/s | 2.0 MB 00:00 2024-04-24T01:53:43.751 INFO:teuthology.orchestra.run.smithi079.stdout:(10/11): python3-routes-2.5.1-5.el9.noarch.rpm 2.2 MB/s | 188 kB 00:00 2024-04-24T01:53:44.035 INFO:teuthology.orchestra.run.smithi079.stdout:(11/11): protobuf-compiler-3.14.0-13.el9.x86_64 1.0 MB/s | 863 kB 00:00 2024-04-24T01:53:44.037 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-04-24T01:53:44.037 INFO:teuthology.orchestra.run.smithi079.stdout:Total 5.7 MB/s | 7.6 MB 00:01 2024-04-24T01:53:44.068 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-04-24T01:53:44.086 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-04-24T01:53:44.086 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-04-24T01:53:44.232 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-04-24T01:53:44.301 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-04-24T01:53:44.716 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-04-24T01:53:44.817 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/11 2024-04-24T01:53:44.854 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/11 2024-04-24T01:53:44.932 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 3/11 2024-04-24T01:53:45.074 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 4/11 2024-04-24T01:53:45.124 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 5/11 2024-04-24T01:53:45.209 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 6/11 2024-04-24T01:53:45.260 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 7/11 2024-04-24T01:53:45.289 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 8/11 2024-04-24T01:53:45.312 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3227.g424d8c43.el9 9/11 2024-04-24T01:53:46.357 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3227.g424d8c43.el 10/11 2024-04-24T01:53:46.413 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3227.g424d8c43.el9.noa 11/11 2024-04-24T01:53:47.785 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3227.g424d8c43.el9.noa 11/11 2024-04-24T01:53:47.785 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3227.g424d8c43.el 1/11 2024-04-24T01:53:47.786 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3227.g424d8c43.el9.noa 2/11 2024-04-24T01:53:47.786 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3227.g424d8c43.el9 3/11 2024-04-24T01:53:47.786 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 4/11 2024-04-24T01:53:47.786 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 5/11 2024-04-24T01:53:47.786 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 6/11 2024-04-24T01:53:47.786 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 7/11 2024-04-24T01:53:47.786 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 8/11 2024-04-24T01:53:47.786 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/11 2024-04-24T01:53:47.786 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 10/11 2024-04-24T01:53:48.069 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 11/11 2024-04-24T01:53:48.069 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:53:48.069 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-04-24T01:53:48.069 INFO:teuthology.orchestra.run.smithi079.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-24T01:53:48.069 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-grafana-dashboards-2:19.0.0-3227.g424d8c43.el9.noarch 2024-04-24T01:53:48.069 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-dashboard-2:19.0.0-3227.g424d8c43.el9.noarch 2024-04-24T01:53:48.069 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-prometheus-alerts-2:19.0.0-3227.g424d8c43.el9.noarch 2024-04-24T01:53:48.070 INFO:teuthology.orchestra.run.smithi079.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-24T01:53:48.070 INFO:teuthology.orchestra.run.smithi079.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-24T01:53:48.070 INFO:teuthology.orchestra.run.smithi079.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-24T01:53:48.070 INFO:teuthology.orchestra.run.smithi079.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-24T01:53:48.070 INFO:teuthology.orchestra.run.smithi079.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-24T01:53:48.070 INFO:teuthology.orchestra.run.smithi079.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-24T01:53:48.070 INFO:teuthology.orchestra.run.smithi079.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-24T01:53:48.070 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:53:48.070 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-04-24T01:53:48.367 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-24T01:53:48.873 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:41 ago on Wed 24 Apr 2024 01:52:07 AM UTC. 2024-04-24T01:53:48.995 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-04-24T01:53:48.996 INFO:teuthology.orchestra.run.smithi079.stdout:======================================================================================= 2024-04-24T01:53:48.996 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2024-04-24T01:53:48.996 INFO:teuthology.orchestra.run.smithi079.stdout:======================================================================================= 2024-04-24T01:53:48.996 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-04-24T01:53:48.996 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3227.g424d8c43.el9 ceph-noarch 7.4 M 2024-04-24T01:53:48.997 INFO:teuthology.orchestra.run.smithi079.stdout:Installing dependencies: 2024-04-24T01:53:48.997 INFO:teuthology.orchestra.run.smithi079.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-24T01:53:48.997 INFO:teuthology.orchestra.run.smithi079.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-24T01:53:48.997 INFO:teuthology.orchestra.run.smithi079.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-24T01:53:48.997 INFO:teuthology.orchestra.run.smithi079.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-24T01:53:48.997 INFO:teuthology.orchestra.run.smithi079.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-24T01:53:48.997 INFO:teuthology.orchestra.run.smithi079.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-24T01:53:48.997 INFO:teuthology.orchestra.run.smithi079.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-24T01:53:48.997 INFO:teuthology.orchestra.run.smithi079.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-24T01:53:48.997 INFO:teuthology.orchestra.run.smithi079.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-24T01:53:48.997 INFO:teuthology.orchestra.run.smithi079.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-24T01:53:48.997 INFO:teuthology.orchestra.run.smithi079.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-24T01:53:48.997 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:53:48.997 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-04-24T01:53:48.997 INFO:teuthology.orchestra.run.smithi079.stdout:======================================================================================= 2024-04-24T01:53:48.998 INFO:teuthology.orchestra.run.smithi079.stdout:Install 12 Packages 2024-04-24T01:53:48.998 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:53:48.998 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 38 M 2024-04-24T01:53:48.998 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 209 M 2024-04-24T01:53:48.998 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-04-24T01:53:49.302 INFO:teuthology.orchestra.run.smithi079.stdout:(1/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.1 MB/s | 195 kB 00:00 2024-04-24T01:53:49.352 INFO:teuthology.orchestra.run.smithi079.stdout:(2/12): libgfortran-11.4.1-3.el9.x86_64.rpm 3.5 MB/s | 803 kB 00:00 2024-04-24T01:53:49.444 INFO:teuthology.orchestra.run.smithi079.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 230 kB/s | 33 kB 00:00 2024-04-24T01:53:49.503 INFO:teuthology.orchestra.run.smithi079.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 312 kB/s | 18 kB 00:00 2024-04-24T01:53:49.561 INFO:teuthology.orchestra.run.smithi079.stdout:(5/12): openblas-0.3.21-2.el9.x86_64.rpm 603 kB/s | 35 kB 00:00 2024-04-24T01:53:49.779 INFO:teuthology.orchestra.run.smithi079.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 7.0 MB/s | 3.0 MB 00:00 2024-04-24T01:53:49.837 INFO:teuthology.orchestra.run.smithi079.stdout:(7/12): python3-devel-3.9.18-3.el9.x86_64.rpm 4.1 MB/s | 244 kB 00:00 2024-04-24T01:53:50.004 INFO:teuthology.orchestra.run.smithi079.stdout:(8/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 11 MB/s | 4.7 MB 00:00 2024-04-24T01:53:50.062 INFO:teuthology.orchestra.run.smithi079.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 4.2 MB/s | 250 kB 00:00 2024-04-24T01:53:50.188 INFO:teuthology.orchestra.run.smithi079.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 15 MB/s | 5.1 MB 00:00 2024-04-24T01:53:50.338 INFO:teuthology.orchestra.run.smithi079.stdout:(11/12): ceph-mgr-diskprediction-local-19.0.0-3 6.1 MB/s | 7.4 MB 00:01 2024-04-24T01:53:50.830 INFO:teuthology.orchestra.run.smithi079.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 21 MB/s | 16 MB 00:00 2024-04-24T01:53:50.831 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-04-24T01:53:50.831 INFO:teuthology.orchestra.run.smithi079.stdout:Total 21 MB/s | 38 MB 00:01 2024-04-24T01:53:51.179 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-04-24T01:53:51.207 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-04-24T01:53:51.207 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-04-24T01:53:51.614 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-04-24T01:53:51.614 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-04-24T01:53:52.426 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-04-24T01:53:52.485 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-04-24T01:53:52.528 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-04-24T01:53:52.637 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-04-24T01:53:52.670 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 4/12 2024-04-24T01:53:52.848 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-24T01:53:52.878 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-24T01:53:52.963 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-04-24T01:53:53.625 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-04-24T01:53:53.750 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-04-24T01:53:54.802 INFO:teuthology.orchestra.run.smithi103.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 772 kB/s | 16 MB 00:21 2024-04-24T01:53:54.803 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-04-24T01:53:54.803 INFO:teuthology.orchestra.run.smithi103.stdout:Total 1.4 MB/s | 38 MB 00:27 2024-04-24T01:53:54.967 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-04-24T01:53:55.151 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-04-24T01:53:55.178 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-04-24T01:53:55.179 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-04-24T01:53:55.585 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-04-24T01:53:55.585 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-04-24T01:53:56.000 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-04-24T01:53:56.036 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3227.g424d8 12/12 2024-04-24T01:53:56.433 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-04-24T01:53:56.515 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-04-24T01:53:56.558 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-04-24T01:53:56.660 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-04-24T01:53:56.693 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 4/12 2024-04-24T01:53:56.870 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-24T01:53:56.900 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-24T01:53:56.977 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-04-24T01:53:57.647 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-04-24T01:53:57.773 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-04-24T01:53:58.623 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3227.g424d8 12/12 2024-04-24T01:53:58.623 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3227.g424d8 1/12 2024-04-24T01:53:58.623 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-04-24T01:53:58.623 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-04-24T01:53:58.623 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-04-24T01:53:58.623 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-04-24T01:53:58.623 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-04-24T01:53:58.623 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-04-24T01:53:58.623 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-04-24T01:53:58.623 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 9/12 2024-04-24T01:53:58.624 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-04-24T01:53:58.624 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-04-24T01:53:59.011 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-04-24T01:53:59.035 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-04-24T01:53:59.035 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:53:59.035 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-04-24T01:53:59.035 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3227.g424d8c43.el9.noarch 2024-04-24T01:53:59.035 INFO:teuthology.orchestra.run.smithi079.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-24T01:53:59.035 INFO:teuthology.orchestra.run.smithi079.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-24T01:53:59.035 INFO:teuthology.orchestra.run.smithi079.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-24T01:53:59.035 INFO:teuthology.orchestra.run.smithi079.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-24T01:53:59.035 INFO:teuthology.orchestra.run.smithi079.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-24T01:53:59.035 INFO:teuthology.orchestra.run.smithi079.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-24T01:53:59.035 INFO:teuthology.orchestra.run.smithi079.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-24T01:53:59.035 INFO:teuthology.orchestra.run.smithi079.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-24T01:53:59.036 INFO:teuthology.orchestra.run.smithi079.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-24T01:53:59.036 INFO:teuthology.orchestra.run.smithi079.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-24T01:53:59.036 INFO:teuthology.orchestra.run.smithi079.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-24T01:53:59.036 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:53:59.036 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-04-24T01:53:59.414 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-mgr-rook 2024-04-24T01:53:59.926 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:52 ago on Wed 24 Apr 2024 01:52:07 AM UTC. 2024-04-24T01:54:00.040 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-04-24T01:54:00.051 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-04-24T01:54:00.053 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-04-24T01:54:00.053 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2024-04-24T01:54:00.053 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-04-24T01:54:00.053 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-04-24T01:54:00.053 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-rook noarch 2:19.0.0-3227.g424d8c43.el9 ceph-noarch 49 k 2024-04-24T01:54:00.053 INFO:teuthology.orchestra.run.smithi079.stdout:Installing dependencies: 2024-04-24T01:54:00.053 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-24T01:54:00.053 INFO:teuthology.orchestra.run.smithi079.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-24T01:54:00.053 INFO:teuthology.orchestra.run.smithi079.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-24T01:54:00.053 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-24T01:54:00.053 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-24T01:54:00.054 INFO:teuthology.orchestra.run.smithi079.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-24T01:54:00.054 INFO:teuthology.orchestra.run.smithi079.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-24T01:54:00.054 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-24T01:54:00.054 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-24T01:54:00.054 INFO:teuthology.orchestra.run.smithi079.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-24T01:54:00.054 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-24T01:54:00.054 INFO:teuthology.orchestra.run.smithi079.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-24T01:54:00.054 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:54:00.054 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-04-24T01:54:00.054 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-04-24T01:54:00.054 INFO:teuthology.orchestra.run.smithi079.stdout:Install 13 Packages 2024-04-24T01:54:00.054 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:54:00.055 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 2.2 M 2024-04-24T01:54:00.055 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 26 M 2024-04-24T01:54:00.055 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-04-24T01:54:00.075 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3227.g424d8 12/12 2024-04-24T01:54:00.444 INFO:teuthology.orchestra.run.smithi079.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 265 kB/s | 26 kB 00:00 2024-04-24T01:54:00.469 INFO:teuthology.orchestra.run.smithi079.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 157 kB/s | 19 kB 00:00 2024-04-24T01:54:00.536 INFO:teuthology.orchestra.run.smithi079.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.4 MB/s | 222 kB 00:00 2024-04-24T01:54:00.586 INFO:teuthology.orchestra.run.smithi079.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.3 MB/s | 159 kB 00:00 2024-04-24T01:54:00.612 INFO:teuthology.orchestra.run.smithi079.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 3.6 MB/s | 279 kB 00:00 2024-04-24T01:54:00.637 INFO:teuthology.orchestra.run.smithi079.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 54 kB 00:00 2024-04-24T01:54:00.654 INFO:teuthology.orchestra.run.smithi079.stdout:(7/13): ceph-mgr-rook-19.0.0-3227.g424d8c43.el9 161 kB/s | 49 kB 00:00 2024-04-24T01:54:00.888 INFO:teuthology.orchestra.run.smithi079.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 117 kB/s | 32 kB 00:00 2024-04-24T01:54:00.913 INFO:teuthology.orchestra.run.smithi079.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 54 kB/s | 15 kB 00:00 2024-04-24T01:54:01.063 INFO:teuthology.orchestra.run.smithi079.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 393 kB/s | 59 kB 00:00 2024-04-24T01:54:01.088 INFO:teuthology.orchestra.run.smithi079.stdout:(11/13): python3-google-auth-2.29.0-1.el9.noarc 473 kB/s | 206 kB 00:00 2024-04-24T01:54:01.189 INFO:teuthology.orchestra.run.smithi079.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 718 kB/s | 90 kB 00:00 2024-04-24T01:54:01.222 INFO:teuthology.orchestra.run.smithi079.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 3.1 MB/s | 1.0 MB 00:00 2024-04-24T01:54:01.225 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-04-24T01:54:01.225 INFO:teuthology.orchestra.run.smithi079.stdout:Total 1.9 MB/s | 2.2 MB 00:01 2024-04-24T01:54:01.285 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-04-24T01:54:01.299 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-04-24T01:54:01.299 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-04-24T01:54:01.382 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-04-24T01:54:01.382 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-04-24T01:54:01.880 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-04-24T01:54:02.044 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-04-24T01:54:02.106 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-04-24T01:54:02.162 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-04-24T01:54:02.199 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-04-24T01:54:02.240 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-04-24T01:54:02.343 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-04-24T01:54:02.445 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 7/13 2024-04-24T01:54:02.500 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-04-24T01:54:02.628 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3227.g424d8 12/12 2024-04-24T01:54:02.628 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3227.g424d8 1/12 2024-04-24T01:54:02.628 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-04-24T01:54:02.628 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-04-24T01:54:02.628 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-04-24T01:54:02.629 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-04-24T01:54:02.629 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-04-24T01:54:02.629 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-04-24T01:54:02.629 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-04-24T01:54:02.629 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 9/12 2024-04-24T01:54:02.629 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-04-24T01:54:02.629 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-04-24T01:54:03.048 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-04-24T01:54:03.049 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:54:03.049 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-04-24T01:54:03.049 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3227.g424d8c43.el9.noarch 2024-04-24T01:54:03.049 INFO:teuthology.orchestra.run.smithi103.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-24T01:54:03.049 INFO:teuthology.orchestra.run.smithi103.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-24T01:54:03.049 INFO:teuthology.orchestra.run.smithi103.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-24T01:54:03.049 INFO:teuthology.orchestra.run.smithi103.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-24T01:54:03.049 INFO:teuthology.orchestra.run.smithi103.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-24T01:54:03.049 INFO:teuthology.orchestra.run.smithi103.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-24T01:54:03.049 INFO:teuthology.orchestra.run.smithi103.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-24T01:54:03.050 INFO:teuthology.orchestra.run.smithi103.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-24T01:54:03.050 INFO:teuthology.orchestra.run.smithi103.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-24T01:54:03.050 INFO:teuthology.orchestra.run.smithi103.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-24T01:54:03.050 INFO:teuthology.orchestra.run.smithi103.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-24T01:54:03.050 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:54:03.050 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-04-24T01:54:03.209 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-04-24T01:54:03.265 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-04-24T01:54:03.298 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-04-24T01:54:03.354 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-04-24T01:54:03.376 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mgr-rook-2:19.0.0-3227.g424d8c43.el9.noarch 13/13 2024-04-24T01:54:03.395 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-mgr-rook 2024-04-24T01:54:03.911 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:02:17 ago on Wed 24 Apr 2024 01:51:46 AM UTC. 2024-04-24T01:54:04.034 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-04-24T01:54:04.035 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-24T01:54:04.036 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repository Size 2024-04-24T01:54:04.036 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-24T01:54:04.036 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-04-24T01:54:04.036 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-rook noarch 2:19.0.0-3227.g424d8c43.el9 ceph-noarch 49 k 2024-04-24T01:54:04.036 INFO:teuthology.orchestra.run.smithi103.stdout:Installing dependencies: 2024-04-24T01:54:04.036 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-24T01:54:04.036 INFO:teuthology.orchestra.run.smithi103.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-24T01:54:04.036 INFO:teuthology.orchestra.run.smithi103.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-24T01:54:04.036 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-24T01:54:04.036 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-24T01:54:04.036 INFO:teuthology.orchestra.run.smithi103.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-24T01:54:04.036 INFO:teuthology.orchestra.run.smithi103.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-24T01:54:04.036 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-24T01:54:04.036 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-24T01:54:04.037 INFO:teuthology.orchestra.run.smithi103.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-24T01:54:04.037 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-24T01:54:04.037 INFO:teuthology.orchestra.run.smithi103.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-24T01:54:04.037 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:54:04.037 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-04-24T01:54:04.037 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-24T01:54:04.037 INFO:teuthology.orchestra.run.smithi103.stdout:Install 13 Packages 2024-04-24T01:54:04.037 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:54:04.038 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 2.2 M 2024-04-24T01:54:04.038 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 26 M 2024-04-24T01:54:04.038 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-04-24T01:54:04.151 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3227.g424d8c43.el9.noarch 13/13 2024-04-24T01:54:04.151 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3227.g424d8c43.el9.noarch 1/13 2024-04-24T01:54:04.151 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-04-24T01:54:04.151 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-04-24T01:54:04.151 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-04-24T01:54:04.151 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-04-24T01:54:04.152 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-04-24T01:54:04.152 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-04-24T01:54:04.152 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-04-24T01:54:04.152 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-04-24T01:54:04.152 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 10/13 2024-04-24T01:54:04.152 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-04-24T01:54:04.152 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-04-24T01:54:04.336 INFO:teuthology.orchestra.run.smithi103.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 234 kB/s | 19 kB 00:00 2024-04-24T01:54:04.362 INFO:teuthology.orchestra.run.smithi103.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 239 kB/s | 26 kB 00:00 2024-04-24T01:54:04.453 INFO:teuthology.orchestra.run.smithi103.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.9 MB/s | 222 kB 00:00 2024-04-24T01:54:04.458 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-04-24T01:54:04.458 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:54:04.459 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-04-24T01:54:04.459 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-rook-2:19.0.0-3227.g424d8c43.el9.noarch 2024-04-24T01:54:04.459 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-24T01:54:04.459 INFO:teuthology.orchestra.run.smithi079.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-24T01:54:04.459 INFO:teuthology.orchestra.run.smithi079.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-24T01:54:04.459 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-24T01:54:04.459 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-24T01:54:04.459 INFO:teuthology.orchestra.run.smithi079.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-24T01:54:04.459 INFO:teuthology.orchestra.run.smithi079.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-24T01:54:04.459 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-24T01:54:04.459 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-24T01:54:04.459 INFO:teuthology.orchestra.run.smithi079.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-24T01:54:04.459 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-24T01:54:04.459 INFO:teuthology.orchestra.run.smithi079.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-24T01:54:04.459 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:54:04.460 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-04-24T01:54:04.479 INFO:teuthology.orchestra.run.smithi103.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.3 MB/s | 159 kB 00:00 2024-04-24T01:54:04.504 INFO:teuthology.orchestra.run.smithi103.stdout:(5/13): ceph-mgr-rook-19.0.0-3227.g424d8c43.el9 197 kB/s | 49 kB 00:00 2024-04-24T01:54:04.529 INFO:teuthology.orchestra.run.smithi103.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 54 kB 00:00 2024-04-24T01:54:04.563 INFO:teuthology.orchestra.run.smithi103.stdout:(7/13): python3-pyasn1-modules-0.4.8-6.el9.noar 2.5 MB/s | 279 kB 00:00 2024-04-24T01:54:04.663 INFO:teuthology.orchestra.run.smithi103.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 202 kB/s | 32 kB 00:00 2024-04-24T01:54:04.680 INFO:teuthology.orchestra.run.smithi103.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 99 kB/s | 15 kB 00:00 2024-04-24T01:54:04.719 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-mgr-cephadm 2024-04-24T01:54:04.747 INFO:teuthology.orchestra.run.smithi103.stdout:(10/13): python3-google-auth-2.29.0-1.el9.noarc 1.5 MB/s | 206 kB 00:00 2024-04-24T01:54:04.797 INFO:teuthology.orchestra.run.smithi103.stdout:(11/13): python3-kubernetes-26.1.0-2.el9.noarch 7.7 MB/s | 1.0 MB 00:00 2024-04-24T01:54:04.823 INFO:teuthology.orchestra.run.smithi103.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 416 kB/s | 59 kB 00:00 2024-04-24T01:54:04.848 INFO:teuthology.orchestra.run.smithi103.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 904 kB/s | 90 kB 00:00 2024-04-24T01:54:04.850 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-04-24T01:54:04.850 INFO:teuthology.orchestra.run.smithi103.stdout:Total 2.7 MB/s | 2.2 MB 00:00 2024-04-24T01:54:04.909 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-04-24T01:54:04.924 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-04-24T01:54:04.924 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-04-24T01:54:05.008 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-04-24T01:54:05.009 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-04-24T01:54:05.226 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:58 ago on Wed 24 Apr 2024 01:52:07 AM UTC. 2024-04-24T01:54:05.345 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-04-24T01:54:05.345 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-04-24T01:54:05.345 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2024-04-24T01:54:05.346 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-04-24T01:54:05.346 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-04-24T01:54:05.346 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3227.g424d8c43.el9 ceph-noarch 142 k 2024-04-24T01:54:05.346 INFO:teuthology.orchestra.run.smithi079.stdout:Installing dependencies: 2024-04-24T01:54:05.346 INFO:teuthology.orchestra.run.smithi079.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-24T01:54:05.346 INFO:teuthology.orchestra.run.smithi079.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-24T01:54:05.346 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-24T01:54:05.346 INFO:teuthology.orchestra.run.smithi079.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-24T01:54:05.346 INFO:teuthology.orchestra.run.smithi079.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-24T01:54:05.346 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:54:05.347 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-04-24T01:54:05.347 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-04-24T01:54:05.347 INFO:teuthology.orchestra.run.smithi079.stdout:Install 6 Packages 2024-04-24T01:54:05.347 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:54:05.347 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 7.0 M 2024-04-24T01:54:05.347 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 33 M 2024-04-24T01:54:05.347 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-04-24T01:54:05.509 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-04-24T01:54:05.687 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-04-24T01:54:05.749 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-04-24T01:54:05.804 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-04-24T01:54:05.825 INFO:teuthology.orchestra.run.smithi079.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.2 MB/s | 248 kB 00:00 2024-04-24T01:54:05.842 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-04-24T01:54:05.850 INFO:teuthology.orchestra.run.smithi079.stdout:(2/6): ceph-mgr-cephadm-19.0.0-3227.g424d8c43.e 620 kB/s | 142 kB 00:00 2024-04-24T01:54:05.882 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-04-24T01:54:05.976 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-04-24T01:54:06.026 INFO:teuthology.orchestra.run.smithi079.stdout:(3/6): python3-babel-2.9.1-2.el9.noarch.rpm 15 MB/s | 6.0 MB 00:00 2024-04-24T01:54:06.051 INFO:teuthology.orchestra.run.smithi079.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 288 kB/s | 58 kB 00:00 2024-04-24T01:54:06.095 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 7/13 2024-04-24T01:54:06.159 INFO:teuthology.orchestra.run.smithi079.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.6 MB/s | 548 kB 00:00 2024-04-24T01:54:06.176 INFO:teuthology.orchestra.run.smithi079.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 339 kB/s | 51 kB 00:00 2024-04-24T01:54:06.177 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-04-24T01:54:06.178 INFO:teuthology.orchestra.run.smithi079.stdout:Total 8.4 MB/s | 7.0 MB 00:00 2024-04-24T01:54:06.218 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-04-24T01:54:06.247 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-04-24T01:54:06.258 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-04-24T01:54:06.258 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-04-24T01:54:06.354 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-04-24T01:54:06.355 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-04-24T01:54:06.660 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-04-24T01:54:06.816 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-04-24T01:54:06.878 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-04-24T01:54:06.933 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-04-24T01:54:06.982 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-04-24T01:54:07.016 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-04-24T01:54:07.063 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-04-24T01:54:07.085 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mgr-rook-2:19.0.0-3227.g424d8c43.el9.noarch 13/13 2024-04-24T01:54:07.392 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-04-24T01:54:07.471 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-04-24T01:54:07.534 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-04-24T01:54:07.573 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3227.g424d8c43.el9.noarch 6/6 2024-04-24T01:54:07.762 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3227.g424d8c43.el9.noarch 13/13 2024-04-24T01:54:07.762 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3227.g424d8c43.el9.noarch 1/13 2024-04-24T01:54:07.762 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-04-24T01:54:07.762 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-04-24T01:54:07.762 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-04-24T01:54:07.762 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-04-24T01:54:07.763 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-04-24T01:54:07.763 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-04-24T01:54:07.763 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-04-24T01:54:07.763 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-04-24T01:54:07.763 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 10/13 2024-04-24T01:54:07.763 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-04-24T01:54:07.763 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-04-24T01:54:08.060 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-04-24T01:54:08.060 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:54:08.060 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-04-24T01:54:08.060 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-rook-2:19.0.0-3227.g424d8c43.el9.noarch 2024-04-24T01:54:08.060 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-24T01:54:08.061 INFO:teuthology.orchestra.run.smithi103.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-24T01:54:08.061 INFO:teuthology.orchestra.run.smithi103.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-24T01:54:08.061 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-24T01:54:08.061 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-24T01:54:08.061 INFO:teuthology.orchestra.run.smithi103.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-24T01:54:08.061 INFO:teuthology.orchestra.run.smithi103.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-24T01:54:08.061 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-24T01:54:08.061 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-24T01:54:08.062 INFO:teuthology.orchestra.run.smithi103.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-24T01:54:08.062 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-24T01:54:08.062 INFO:teuthology.orchestra.run.smithi103.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-24T01:54:08.062 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:54:08.062 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-04-24T01:54:08.345 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-mgr-cephadm 2024-04-24T01:54:08.383 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3227.g424d8c43.el9.noarch 6/6 2024-04-24T01:54:08.383 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3227.g424d8c43.el9.noarch 1/6 2024-04-24T01:54:08.383 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-04-24T01:54:08.383 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-04-24T01:54:08.383 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-04-24T01:54:08.383 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-04-24T01:54:08.648 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-04-24T01:54:08.648 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:54:08.648 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-04-24T01:54:08.648 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-cephadm-2:19.0.0-3227.g424d8c43.el9.noarch 2024-04-24T01:54:08.649 INFO:teuthology.orchestra.run.smithi079.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-24T01:54:08.649 INFO:teuthology.orchestra.run.smithi079.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-24T01:54:08.649 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-24T01:54:08.649 INFO:teuthology.orchestra.run.smithi079.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-24T01:54:08.649 INFO:teuthology.orchestra.run.smithi079.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-24T01:54:08.649 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:54:08.649 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-04-24T01:54:08.856 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:02:22 ago on Wed 24 Apr 2024 01:51:46 AM UTC. 2024-04-24T01:54:08.935 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-fuse 2024-04-24T01:54:08.975 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-04-24T01:54:08.975 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-24T01:54:08.976 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repository Size 2024-04-24T01:54:08.976 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-24T01:54:08.976 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-04-24T01:54:08.976 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3227.g424d8c43.el9 ceph-noarch 142 k 2024-04-24T01:54:08.976 INFO:teuthology.orchestra.run.smithi103.stdout:Installing dependencies: 2024-04-24T01:54:08.976 INFO:teuthology.orchestra.run.smithi103.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-24T01:54:08.976 INFO:teuthology.orchestra.run.smithi103.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-24T01:54:08.976 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-24T01:54:08.976 INFO:teuthology.orchestra.run.smithi103.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-24T01:54:08.976 INFO:teuthology.orchestra.run.smithi103.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-24T01:54:08.976 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:54:08.976 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-04-24T01:54:08.976 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-24T01:54:08.977 INFO:teuthology.orchestra.run.smithi103.stdout:Install 6 Packages 2024-04-24T01:54:08.977 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:54:08.977 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 7.0 M 2024-04-24T01:54:08.977 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 33 M 2024-04-24T01:54:08.977 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-04-24T01:54:09.433 INFO:teuthology.orchestra.run.smithi103.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.3 MB/s | 248 kB 00:00 2024-04-24T01:54:09.451 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:02:02 ago on Wed 24 Apr 2024 01:52:07 AM UTC. 2024-04-24T01:54:09.571 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-04-24T01:54:09.572 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-04-24T01:54:09.572 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2024-04-24T01:54:09.572 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-04-24T01:54:09.572 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-04-24T01:54:09.572 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-fuse x86_64 2:19.0.0-3227.g424d8c43.el9 ceph 871 k 2024-04-24T01:54:09.572 INFO:teuthology.orchestra.run.smithi079.stdout:Installing dependencies: 2024-04-24T01:54:09.572 INFO:teuthology.orchestra.run.smithi079.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-24T01:54:09.572 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:54:09.572 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-04-24T01:54:09.572 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-04-24T01:54:09.572 INFO:teuthology.orchestra.run.smithi079.stdout:Install 2 Packages 2024-04-24T01:54:09.572 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:54:09.573 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 951 k 2024-04-24T01:54:09.573 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 4.0 M 2024-04-24T01:54:09.573 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-04-24T01:54:09.592 INFO:teuthology.orchestra.run.smithi103.stdout:(2/6): ceph-mgr-cephadm-19.0.0-3227.g424d8c43.e 419 kB/s | 142 kB 00:00 2024-04-24T01:54:09.775 INFO:teuthology.orchestra.run.smithi103.stdout:(3/6): python3-babel-2.9.1-2.el9.noarch.rpm 11 MB/s | 6.0 MB 00:00 2024-04-24T01:54:09.786 INFO:teuthology.orchestra.run.smithi079.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 527 kB/s | 80 kB 00:00 2024-04-24T01:54:09.950 INFO:teuthology.orchestra.run.smithi103.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 161 kB/s | 58 kB 00:00 2024-04-24T01:54:09.976 INFO:teuthology.orchestra.run.smithi103.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.0 MB/s | 548 kB 00:00 2024-04-24T01:54:10.059 INFO:teuthology.orchestra.run.smithi103.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 180 kB/s | 51 kB 00:00 2024-04-24T01:54:10.062 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-04-24T01:54:10.062 INFO:teuthology.orchestra.run.smithi103.stdout:Total 6.4 MB/s | 7.0 MB 00:01 2024-04-24T01:54:10.078 INFO:teuthology.orchestra.run.smithi079.stdout:(2/2): ceph-fuse-19.0.0-3227.g424d8c43.el9.x86_ 1.9 MB/s | 871 kB 00:00 2024-04-24T01:54:10.079 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-04-24T01:54:10.079 INFO:teuthology.orchestra.run.smithi079.stdout:Total 1.8 MB/s | 951 kB 00:00 2024-04-24T01:54:10.090 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-04-24T01:54:10.115 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-04-24T01:54:10.115 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-04-24T01:54:10.132 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-04-24T01:54:10.143 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-04-24T01:54:10.143 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-04-24T01:54:10.193 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-04-24T01:54:10.194 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-04-24T01:54:10.239 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-04-24T01:54:10.241 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-04-24T01:54:10.372 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-04-24T01:54:10.515 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-04-24T01:54:10.541 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-04-24T01:54:10.578 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-fuse-2:19.0.0-3227.g424d8c43.el9.x86_64 2/2 2024-04-24T01:54:10.726 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-04-24T01:54:10.777 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-04-24T01:54:11.230 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-fuse-2:19.0.0-3227.g424d8c43.el9.x86_64 2/2 2024-04-24T01:54:11.230 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-fuse-2:19.0.0-3227.g424d8c43.el9.x86_64 1/2 2024-04-24T01:54:11.313 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-04-24T01:54:11.390 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-04-24T01:54:11.461 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-04-24T01:54:11.488 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-04-24T01:54:11.488 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:54:11.488 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-04-24T01:54:11.488 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-fuse-2:19.0.0-3227.g424d8c43.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-04-24T01:54:11.488 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:54:11.488 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-04-24T01:54:11.490 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3227.g424d8c43.el9.noarch 6/6 2024-04-24T01:54:11.786 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-volume 2024-04-24T01:54:12.302 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:02:05 ago on Wed 24 Apr 2024 01:52:07 AM UTC. 2024-04-24T01:54:12.309 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3227.g424d8c43.el9.noarch 6/6 2024-04-24T01:54:12.309 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3227.g424d8c43.el9.noarch 1/6 2024-04-24T01:54:12.309 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-04-24T01:54:12.309 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-04-24T01:54:12.309 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-04-24T01:54:12.309 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-04-24T01:54:12.428 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-04-24T01:54:12.429 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-04-24T01:54:12.429 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2024-04-24T01:54:12.429 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-04-24T01:54:12.429 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-04-24T01:54:12.429 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-volume noarch 2:19.0.0-3227.g424d8c43.el9 ceph-noarch 273 k 2024-04-24T01:54:12.429 INFO:teuthology.orchestra.run.smithi079.stdout:Installing dependencies: 2024-04-24T01:54:12.429 INFO:teuthology.orchestra.run.smithi079.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-24T01:54:12.429 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:54:12.429 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-04-24T01:54:12.429 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-04-24T01:54:12.429 INFO:teuthology.orchestra.run.smithi079.stdout:Install 2 Packages 2024-04-24T01:54:12.430 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:54:12.430 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 350 k 2024-04-24T01:54:12.430 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 1.5 M 2024-04-24T01:54:12.430 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-04-24T01:54:12.567 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-04-24T01:54:12.567 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:54:12.568 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-04-24T01:54:12.568 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-cephadm-2:19.0.0-3227.g424d8c43.el9.noarch 2024-04-24T01:54:12.568 INFO:teuthology.orchestra.run.smithi103.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-24T01:54:12.568 INFO:teuthology.orchestra.run.smithi103.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-24T01:54:12.568 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-24T01:54:12.568 INFO:teuthology.orchestra.run.smithi103.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-24T01:54:12.568 INFO:teuthology.orchestra.run.smithi103.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-24T01:54:12.568 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:54:12.568 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-04-24T01:54:12.741 INFO:teuthology.orchestra.run.smithi079.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 413 kB/s | 77 kB 00:00 2024-04-24T01:54:12.822 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-fuse 2024-04-24T01:54:12.900 INFO:teuthology.orchestra.run.smithi079.stdout:(2/2): ceph-volume-19.0.0-3227.g424d8c43.el9.no 790 kB/s | 273 kB 00:00 2024-04-24T01:54:12.901 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-04-24T01:54:12.901 INFO:teuthology.orchestra.run.smithi079.stdout:Total 742 kB/s | 350 kB 00:00 2024-04-24T01:54:12.912 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-04-24T01:54:12.922 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-04-24T01:54:12.922 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-04-24T01:54:12.947 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-04-24T01:54:12.948 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-04-24T01:54:13.107 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-04-24T01:54:13.289 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-04-24T01:54:13.319 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-volume-2:19.0.0-3227.g424d8c43.el9.noarch 2/2 2024-04-24T01:54:13.334 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-volume-2:19.0.0-3227.g424d8c43.el9.noarch 2/2 2024-04-24T01:54:13.334 INFO:teuthology.orchestra.run.smithi079.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T01:54:13.334 INFO:teuthology.orchestra.run.smithi079.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-24T01:54:13.335 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:54:13.337 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:02:27 ago on Wed 24 Apr 2024 01:51:46 AM UTC. 2024-04-24T01:54:13.459 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-04-24T01:54:13.459 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-24T01:54:13.459 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repository Size 2024-04-24T01:54:13.459 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-24T01:54:13.459 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-04-24T01:54:13.460 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-fuse x86_64 2:19.0.0-3227.g424d8c43.el9 ceph 871 k 2024-04-24T01:54:13.460 INFO:teuthology.orchestra.run.smithi103.stdout:Installing dependencies: 2024-04-24T01:54:13.460 INFO:teuthology.orchestra.run.smithi103.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-24T01:54:13.460 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:54:13.460 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-04-24T01:54:13.460 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-24T01:54:13.460 INFO:teuthology.orchestra.run.smithi103.stdout:Install 2 Packages 2024-04-24T01:54:13.460 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:54:13.461 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 951 k 2024-04-24T01:54:13.461 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 4.0 M 2024-04-24T01:54:13.461 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-04-24T01:54:13.870 INFO:teuthology.orchestra.run.smithi103.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 232 kB/s | 80 kB 00:00 2024-04-24T01:54:13.986 INFO:teuthology.orchestra.run.smithi103.stdout:(2/2): ceph-fuse-19.0.0-3227.g424d8c43.el9.x86_ 1.8 MB/s | 871 kB 00:00 2024-04-24T01:54:13.987 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-04-24T01:54:13.987 INFO:teuthology.orchestra.run.smithi103.stdout:Total 1.8 MB/s | 951 kB 00:00 2024-04-24T01:54:13.991 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-volume-2:19.0.0-3227.g424d8c43.el9.noarch 1/2 2024-04-24T01:54:13.999 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-04-24T01:54:14.023 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-04-24T01:54:14.023 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-04-24T01:54:14.100 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-04-24T01:54:14.100 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-04-24T01:54:14.285 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-04-24T01:54:14.289 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-04-24T01:54:14.289 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:54:14.289 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-04-24T01:54:14.289 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-volume-2:19.0.0-3227.g424d8c43.el9.noarch 2024-04-24T01:54:14.289 INFO:teuthology.orchestra.run.smithi079.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-24T01:54:14.289 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:54:14.289 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-04-24T01:54:14.421 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-04-24T01:54:14.492 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-fuse-2:19.0.0-3227.g424d8c43.el9.x86_64 2/2 2024-04-24T01:54:14.623 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install librados-devel 2024-04-24T01:54:15.127 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-fuse-2:19.0.0-3227.g424d8c43.el9.x86_64 2/2 2024-04-24T01:54:15.266 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-fuse-2:19.0.0-3227.g424d8c43.el9.x86_64 1/2 2024-04-24T01:54:15.267 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:02:08 ago on Wed 24 Apr 2024 01:52:07 AM UTC. 2024-04-24T01:54:15.270 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-04-24T01:54:15.270 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-04-24T01:54:15.270 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repo Size 2024-04-24T01:54:15.270 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-04-24T01:54:15.271 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-04-24T01:54:15.271 INFO:teuthology.orchestra.run.smithi079.stdout: librados-devel x86_64 2:19.0.0-3227.g424d8c43.el9 ceph 125 k 2024-04-24T01:54:15.271 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:54:15.271 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-04-24T01:54:15.271 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-04-24T01:54:15.271 INFO:teuthology.orchestra.run.smithi079.stdout:Install 1 Package 2024-04-24T01:54:15.271 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:54:15.272 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 125 k 2024-04-24T01:54:15.272 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 445 k 2024-04-24T01:54:15.272 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-04-24T01:54:15.376 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-04-24T01:54:15.376 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:54:15.376 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-04-24T01:54:15.376 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-fuse-2:19.0.0-3227.g424d8c43.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-04-24T01:54:15.376 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:54:15.377 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-04-24T01:54:15.472 INFO:teuthology.orchestra.run.smithi079.stdout:librados-devel-19.0.0-3227.g424d8c43.el9.x86_64 621 kB/s | 125 kB 00:00 2024-04-24T01:54:15.473 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-04-24T01:54:15.473 INFO:teuthology.orchestra.run.smithi079.stdout:Total 617 kB/s | 125 kB 00:00 2024-04-24T01:54:15.473 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-04-24T01:54:15.478 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-04-24T01:54:15.478 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-04-24T01:54:15.541 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-04-24T01:54:15.541 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-04-24T01:54:15.682 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-volume 2024-04-24T01:54:15.698 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-04-24T01:54:15.802 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : librados-devel-2:19.0.0-3227.g424d8c43.el9.x86_64 1/1 2024-04-24T01:54:16.205 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:02:30 ago on Wed 24 Apr 2024 01:51:46 AM UTC. 2024-04-24T01:54:16.220 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: librados-devel-2:19.0.0-3227.g424d8c43.el9.x86_64 1/1 2024-04-24T01:54:16.326 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-04-24T01:54:16.327 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-24T01:54:16.327 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repository Size 2024-04-24T01:54:16.327 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-24T01:54:16.327 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-04-24T01:54:16.327 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-volume noarch 2:19.0.0-3227.g424d8c43.el9 ceph-noarch 273 k 2024-04-24T01:54:16.327 INFO:teuthology.orchestra.run.smithi103.stdout:Installing dependencies: 2024-04-24T01:54:16.327 INFO:teuthology.orchestra.run.smithi103.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-24T01:54:16.327 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:54:16.327 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-04-24T01:54:16.327 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-24T01:54:16.327 INFO:teuthology.orchestra.run.smithi103.stdout:Install 2 Packages 2024-04-24T01:54:16.328 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:54:16.328 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 350 k 2024-04-24T01:54:16.328 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 1.5 M 2024-04-24T01:54:16.328 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-04-24T01:54:16.454 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librados-devel-2:19.0.0-3227.g424d8c43.el9.x86_64 1/1 2024-04-24T01:54:16.455 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:54:16.455 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-04-24T01:54:16.455 INFO:teuthology.orchestra.run.smithi079.stdout: librados-devel-2:19.0.0-3227.g424d8c43.el9.x86_64 2024-04-24T01:54:16.455 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:54:16.455 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-04-24T01:54:16.617 INFO:teuthology.orchestra.run.smithi103.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 528 kB/s | 77 kB 00:00 2024-04-24T01:54:16.673 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install libcephfs2 2024-04-24T01:54:16.825 INFO:teuthology.orchestra.run.smithi103.stdout:(2/2): ceph-volume-19.0.0-3227.g424d8c43.el9.no 770 kB/s | 273 kB 00:00 2024-04-24T01:54:16.825 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-04-24T01:54:16.826 INFO:teuthology.orchestra.run.smithi103.stdout:Total 703 kB/s | 350 kB 00:00 2024-04-24T01:54:16.838 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-04-24T01:54:16.848 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-04-24T01:54:16.848 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-04-24T01:54:16.875 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-04-24T01:54:16.875 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-04-24T01:54:17.036 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-04-24T01:54:17.186 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:02:10 ago on Wed 24 Apr 2024 01:52:07 AM UTC. 2024-04-24T01:54:17.223 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-04-24T01:54:17.271 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-volume-2:19.0.0-3227.g424d8c43.el9.noarch 2/2 2024-04-24T01:54:17.271 INFO:teuthology.orchestra.run.smithi079.stdout:Package libcephfs2-2:19.0.0-3227.g424d8c43.el9.x86_64 is already installed. 2024-04-24T01:54:17.284 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-volume-2:19.0.0-3227.g424d8c43.el9.noarch 2/2 2024-04-24T01:54:17.285 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T01:54:17.285 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-24T01:54:17.285 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:54:17.310 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-04-24T01:54:17.311 INFO:teuthology.orchestra.run.smithi079.stdout:Nothing to do. 2024-04-24T01:54:17.311 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-04-24T01:54:17.411 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install libcephfs-devel 2024-04-24T01:54:17.927 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:02:10 ago on Wed 24 Apr 2024 01:52:07 AM UTC. 2024-04-24T01:54:17.946 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-volume-2:19.0.0-3227.g424d8c43.el9.noarch 1/2 2024-04-24T01:54:18.047 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-04-24T01:54:18.048 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-04-24T01:54:18.048 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repo Size 2024-04-24T01:54:18.048 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-04-24T01:54:18.048 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-04-24T01:54:18.048 INFO:teuthology.orchestra.run.smithi079.stdout: libcephfs-devel x86_64 2:19.0.0-3227.g424d8c43.el9 ceph 32 k 2024-04-24T01:54:18.048 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:54:18.048 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-04-24T01:54:18.048 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-04-24T01:54:18.048 INFO:teuthology.orchestra.run.smithi079.stdout:Install 1 Package 2024-04-24T01:54:18.048 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:54:18.048 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 32 k 2024-04-24T01:54:18.049 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 142 k 2024-04-24T01:54:18.049 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-04-24T01:54:18.226 INFO:teuthology.orchestra.run.smithi079.stdout:libcephfs-devel-19.0.0-3227.g424d8c43.el9.x86_6 178 kB/s | 32 kB 00:00 2024-04-24T01:54:18.227 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-04-24T01:54:18.227 INFO:teuthology.orchestra.run.smithi079.stdout:Total 177 kB/s | 32 kB 00:00 2024-04-24T01:54:18.227 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-04-24T01:54:18.230 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-04-24T01:54:18.230 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-04-24T01:54:18.233 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-04-24T01:54:18.233 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-04-24T01:54:18.235 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-04-24T01:54:18.235 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:54:18.235 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-04-24T01:54:18.235 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-volume-2:19.0.0-3227.g424d8c43.el9.noarch 2024-04-24T01:54:18.235 INFO:teuthology.orchestra.run.smithi103.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-24T01:54:18.235 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:54:18.235 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-04-24T01:54:18.311 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-04-24T01:54:18.413 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libcephfs-devel-2:19.0.0-3227.g424d8c43.el9.x86_64 1/1 2024-04-24T01:54:18.547 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install librados-devel 2024-04-24T01:54:18.749 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-3227.g424d8c43.el9.x86_64 1/1 2024-04-24T01:54:18.988 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libcephfs-devel-2:19.0.0-3227.g424d8c43.el9.x86_64 1/1 2024-04-24T01:54:18.988 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:54:18.988 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-04-24T01:54:18.988 INFO:teuthology.orchestra.run.smithi079.stdout: libcephfs-devel-2:19.0.0-3227.g424d8c43.el9.x86_64 2024-04-24T01:54:18.988 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:54:18.988 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-04-24T01:54:19.073 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:02:33 ago on Wed 24 Apr 2024 01:51:46 AM UTC. 2024-04-24T01:54:19.178 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install librados2 2024-04-24T01:54:19.197 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-04-24T01:54:19.198 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-24T01:54:19.198 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repo Size 2024-04-24T01:54:19.198 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-24T01:54:19.198 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-04-24T01:54:19.198 INFO:teuthology.orchestra.run.smithi103.stdout: librados-devel x86_64 2:19.0.0-3227.g424d8c43.el9 ceph 125 k 2024-04-24T01:54:19.198 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:54:19.198 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-04-24T01:54:19.198 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-24T01:54:19.198 INFO:teuthology.orchestra.run.smithi103.stdout:Install 1 Package 2024-04-24T01:54:19.198 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:54:19.199 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 125 k 2024-04-24T01:54:19.199 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 445 k 2024-04-24T01:54:19.199 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-04-24T01:54:19.423 INFO:teuthology.orchestra.run.smithi103.stdout:librados-devel-19.0.0-3227.g424d8c43.el9.x86_64 558 kB/s | 125 kB 00:00 2024-04-24T01:54:19.423 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-04-24T01:54:19.424 INFO:teuthology.orchestra.run.smithi103.stdout:Total 554 kB/s | 125 kB 00:00 2024-04-24T01:54:19.424 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-04-24T01:54:19.430 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-04-24T01:54:19.430 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-04-24T01:54:19.493 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-04-24T01:54:19.493 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-04-24T01:54:19.646 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-04-24T01:54:19.687 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:02:12 ago on Wed 24 Apr 2024 01:52:07 AM UTC. 2024-04-24T01:54:19.751 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : librados-devel-2:19.0.0-3227.g424d8c43.el9.x86_64 1/1 2024-04-24T01:54:19.767 INFO:teuthology.orchestra.run.smithi079.stdout:Package librados2-2:19.0.0-3227.g424d8c43.el9.x86_64 is already installed. 2024-04-24T01:54:19.809 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-04-24T01:54:19.810 INFO:teuthology.orchestra.run.smithi079.stdout:Nothing to do. 2024-04-24T01:54:19.810 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-04-24T01:54:19.890 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install librbd1 2024-04-24T01:54:20.159 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: librados-devel-2:19.0.0-3227.g424d8c43.el9.x86_64 1/1 2024-04-24T01:54:20.383 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librados-devel-2:19.0.0-3227.g424d8c43.el9.x86_64 1/1 2024-04-24T01:54:20.383 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:54:20.383 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-04-24T01:54:20.383 INFO:teuthology.orchestra.run.smithi103.stdout: librados-devel-2:19.0.0-3227.g424d8c43.el9.x86_64 2024-04-24T01:54:20.383 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:54:20.384 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-04-24T01:54:20.399 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:02:13 ago on Wed 24 Apr 2024 01:52:07 AM UTC. 2024-04-24T01:54:20.478 INFO:teuthology.orchestra.run.smithi079.stdout:Package librbd1-2:19.0.0-3227.g424d8c43.el9.x86_64 is already installed. 2024-04-24T01:54:20.519 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-04-24T01:54:20.520 INFO:teuthology.orchestra.run.smithi079.stdout:Nothing to do. 2024-04-24T01:54:20.521 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-04-24T01:54:20.602 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install python3-rados 2024-04-24T01:54:20.633 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install libcephfs2 2024-04-24T01:54:21.112 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:02:14 ago on Wed 24 Apr 2024 01:52:07 AM UTC. 2024-04-24T01:54:21.142 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:02:35 ago on Wed 24 Apr 2024 01:51:46 AM UTC. 2024-04-24T01:54:21.192 INFO:teuthology.orchestra.run.smithi079.stdout:Package python3-rados-2:19.0.0-3227.g424d8c43.el9.x86_64 is already installed. 2024-04-24T01:54:21.221 INFO:teuthology.orchestra.run.smithi103.stdout:Package libcephfs2-2:19.0.0-3227.g424d8c43.el9.x86_64 is already installed. 2024-04-24T01:54:21.233 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-04-24T01:54:21.235 INFO:teuthology.orchestra.run.smithi079.stdout:Nothing to do. 2024-04-24T01:54:21.235 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-04-24T01:54:21.262 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-04-24T01:54:21.264 INFO:teuthology.orchestra.run.smithi103.stdout:Nothing to do. 2024-04-24T01:54:21.264 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-04-24T01:54:21.314 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install python3-rgw 2024-04-24T01:54:21.355 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install libcephfs-devel 2024-04-24T01:54:21.818 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:02:14 ago on Wed 24 Apr 2024 01:52:07 AM UTC. 2024-04-24T01:54:21.888 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:02:35 ago on Wed 24 Apr 2024 01:51:46 AM UTC. 2024-04-24T01:54:21.896 INFO:teuthology.orchestra.run.smithi079.stdout:Package python3-rgw-2:19.0.0-3227.g424d8c43.el9.x86_64 is already installed. 2024-04-24T01:54:21.936 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-04-24T01:54:21.937 INFO:teuthology.orchestra.run.smithi079.stdout:Nothing to do. 2024-04-24T01:54:21.937 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-04-24T01:54:22.007 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-04-24T01:54:22.008 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-24T01:54:22.008 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repo Size 2024-04-24T01:54:22.008 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-24T01:54:22.008 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-04-24T01:54:22.008 INFO:teuthology.orchestra.run.smithi103.stdout: libcephfs-devel x86_64 2:19.0.0-3227.g424d8c43.el9 ceph 32 k 2024-04-24T01:54:22.008 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:54:22.009 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-04-24T01:54:22.009 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-24T01:54:22.009 INFO:teuthology.orchestra.run.smithi103.stdout:Install 1 Package 2024-04-24T01:54:22.009 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:54:22.009 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 32 k 2024-04-24T01:54:22.009 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 142 k 2024-04-24T01:54:22.009 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-04-24T01:54:22.021 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install python3-cephfs 2024-04-24T01:54:22.251 INFO:teuthology.orchestra.run.smithi103.stdout:libcephfs-devel-19.0.0-3227.g424d8c43.el9.x86_6 131 kB/s | 32 kB 00:00 2024-04-24T01:54:22.252 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-04-24T01:54:22.252 INFO:teuthology.orchestra.run.smithi103.stdout:Total 130 kB/s | 32 kB 00:00 2024-04-24T01:54:22.252 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-04-24T01:54:22.256 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-04-24T01:54:22.256 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-04-24T01:54:22.258 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-04-24T01:54:22.258 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-04-24T01:54:22.335 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-04-24T01:54:22.438 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libcephfs-devel-2:19.0.0-3227.g424d8c43.el9.x86_64 1/1 2024-04-24T01:54:22.535 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:02:15 ago on Wed 24 Apr 2024 01:52:07 AM UTC. 2024-04-24T01:54:22.613 INFO:teuthology.orchestra.run.smithi079.stdout:Package python3-cephfs-2:19.0.0-3227.g424d8c43.el9.x86_64 is already installed. 2024-04-24T01:54:22.655 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-04-24T01:54:22.656 INFO:teuthology.orchestra.run.smithi079.stdout:Nothing to do. 2024-04-24T01:54:22.656 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-04-24T01:54:22.734 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install python3-rbd 2024-04-24T01:54:22.797 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-3227.g424d8c43.el9.x86_64 1/1 2024-04-24T01:54:22.990 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libcephfs-devel-2:19.0.0-3227.g424d8c43.el9.x86_64 1/1 2024-04-24T01:54:22.990 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:54:22.990 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-04-24T01:54:22.990 INFO:teuthology.orchestra.run.smithi103.stdout: libcephfs-devel-2:19.0.0-3227.g424d8c43.el9.x86_64 2024-04-24T01:54:22.991 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:54:22.991 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-04-24T01:54:23.164 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install librados2 2024-04-24T01:54:23.240 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:02:16 ago on Wed 24 Apr 2024 01:52:07 AM UTC. 2024-04-24T01:54:23.318 INFO:teuthology.orchestra.run.smithi079.stdout:Package python3-rbd-2:19.0.0-3227.g424d8c43.el9.x86_64 is already installed. 2024-04-24T01:54:23.358 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-04-24T01:54:23.360 INFO:teuthology.orchestra.run.smithi079.stdout:Nothing to do. 2024-04-24T01:54:23.360 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-04-24T01:54:23.438 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install rbd-fuse 2024-04-24T01:54:23.679 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:02:37 ago on Wed 24 Apr 2024 01:51:46 AM UTC. 2024-04-24T01:54:23.758 INFO:teuthology.orchestra.run.smithi103.stdout:Package librados2-2:19.0.0-3227.g424d8c43.el9.x86_64 is already installed. 2024-04-24T01:54:23.802 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-04-24T01:54:23.802 INFO:teuthology.orchestra.run.smithi103.stdout:Nothing to do. 2024-04-24T01:54:23.802 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-04-24T01:54:23.872 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install librbd1 2024-04-24T01:54:23.951 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:02:16 ago on Wed 24 Apr 2024 01:52:07 AM UTC. 2024-04-24T01:54:24.071 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-04-24T01:54:24.071 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-04-24T01:54:24.072 INFO:teuthology.orchestra.run.smithi079.stdout: Package Architecture Version Repository Size 2024-04-24T01:54:24.072 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-04-24T01:54:24.072 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-04-24T01:54:24.072 INFO:teuthology.orchestra.run.smithi079.stdout: rbd-fuse x86_64 2:19.0.0-3227.g424d8c43.el9 ceph 90 k 2024-04-24T01:54:24.072 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:54:24.072 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-04-24T01:54:24.072 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-04-24T01:54:24.072 INFO:teuthology.orchestra.run.smithi079.stdout:Install 1 Package 2024-04-24T01:54:24.072 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:54:24.072 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 90 k 2024-04-24T01:54:24.072 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 313 k 2024-04-24T01:54:24.072 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-04-24T01:54:24.382 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:02:38 ago on Wed 24 Apr 2024 01:51:46 AM UTC. 2024-04-24T01:54:24.387 INFO:teuthology.orchestra.run.smithi079.stdout:rbd-fuse-19.0.0-3227.g424d8c43.el9.x86_64.rpm 285 kB/s | 90 kB 00:00 2024-04-24T01:54:24.387 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-04-24T01:54:24.388 INFO:teuthology.orchestra.run.smithi079.stdout:Total 284 kB/s | 90 kB 00:00 2024-04-24T01:54:24.388 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-04-24T01:54:24.397 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-04-24T01:54:24.397 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-04-24T01:54:24.460 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-04-24T01:54:24.460 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-04-24T01:54:24.463 INFO:teuthology.orchestra.run.smithi103.stdout:Package librbd1-2:19.0.0-3227.g424d8c43.el9.x86_64 is already installed. 2024-04-24T01:54:24.505 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-04-24T01:54:24.506 INFO:teuthology.orchestra.run.smithi103.stdout:Nothing to do. 2024-04-24T01:54:24.506 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-04-24T01:54:24.569 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install python3-rados 2024-04-24T01:54:24.600 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-04-24T01:54:24.700 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : rbd-fuse-2:19.0.0-3227.g424d8c43.el9.x86_64 1/1 2024-04-24T01:54:25.072 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:02:39 ago on Wed 24 Apr 2024 01:51:46 AM UTC. 2024-04-24T01:54:25.096 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: rbd-fuse-2:19.0.0-3227.g424d8c43.el9.x86_64 1/1 2024-04-24T01:54:25.150 INFO:teuthology.orchestra.run.smithi103.stdout:Package python3-rados-2:19.0.0-3227.g424d8c43.el9.x86_64 is already installed. 2024-04-24T01:54:25.191 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-04-24T01:54:25.192 INFO:teuthology.orchestra.run.smithi103.stdout:Nothing to do. 2024-04-24T01:54:25.193 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-04-24T01:54:25.263 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install python3-rgw 2024-04-24T01:54:25.371 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : rbd-fuse-2:19.0.0-3227.g424d8c43.el9.x86_64 1/1 2024-04-24T01:54:25.371 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:54:25.371 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-04-24T01:54:25.371 INFO:teuthology.orchestra.run.smithi079.stdout: rbd-fuse-2:19.0.0-3227.g424d8c43.el9.x86_64 2024-04-24T01:54:25.371 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:54:25.371 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-04-24T01:54:25.639 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install rbd-mirror 2024-04-24T01:54:25.767 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:02:39 ago on Wed 24 Apr 2024 01:51:46 AM UTC. 2024-04-24T01:54:25.846 INFO:teuthology.orchestra.run.smithi103.stdout:Package python3-rgw-2:19.0.0-3227.g424d8c43.el9.x86_64 is already installed. 2024-04-24T01:54:25.887 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-04-24T01:54:25.888 INFO:teuthology.orchestra.run.smithi103.stdout:Nothing to do. 2024-04-24T01:54:25.888 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-04-24T01:54:25.951 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install python3-cephfs 2024-04-24T01:54:26.149 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:02:19 ago on Wed 24 Apr 2024 01:52:07 AM UTC. 2024-04-24T01:54:26.268 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-04-24T01:54:26.269 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-04-24T01:54:26.269 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repo Size 2024-04-24T01:54:26.269 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-04-24T01:54:26.269 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-04-24T01:54:26.269 INFO:teuthology.orchestra.run.smithi079.stdout: rbd-mirror x86_64 2:19.0.0-3227.g424d8c43.el9 ceph 3.4 M 2024-04-24T01:54:26.269 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:54:26.270 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-04-24T01:54:26.270 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-04-24T01:54:26.270 INFO:teuthology.orchestra.run.smithi079.stdout:Install 1 Package 2024-04-24T01:54:26.270 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:54:26.270 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 3.4 M 2024-04-24T01:54:26.270 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 29 M 2024-04-24T01:54:26.270 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-04-24T01:54:26.462 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:02:40 ago on Wed 24 Apr 2024 01:51:46 AM UTC. 2024-04-24T01:54:26.541 INFO:teuthology.orchestra.run.smithi103.stdout:Package python3-cephfs-2:19.0.0-3227.g424d8c43.el9.x86_64 is already installed. 2024-04-24T01:54:26.582 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-04-24T01:54:26.582 INFO:teuthology.orchestra.run.smithi103.stdout:Nothing to do. 2024-04-24T01:54:26.583 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-04-24T01:54:26.655 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install python3-rbd 2024-04-24T01:54:26.907 INFO:teuthology.orchestra.run.smithi079.stdout:rbd-mirror-19.0.0-3227.g424d8c43.el9.x86_64.rpm 5.3 MB/s | 3.4 MB 00:00 2024-04-24T01:54:26.908 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-04-24T01:54:26.908 INFO:teuthology.orchestra.run.smithi079.stdout:Total 5.3 MB/s | 3.4 MB 00:00 2024-04-24T01:54:26.909 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-04-24T01:54:26.919 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-04-24T01:54:26.920 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-04-24T01:54:27.002 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-04-24T01:54:27.002 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-04-24T01:54:27.168 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:02:41 ago on Wed 24 Apr 2024 01:51:46 AM UTC. 2024-04-24T01:54:27.247 INFO:teuthology.orchestra.run.smithi103.stdout:Package python3-rbd-2:19.0.0-3227.g424d8c43.el9.x86_64 is already installed. 2024-04-24T01:54:27.288 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-04-24T01:54:27.289 INFO:teuthology.orchestra.run.smithi103.stdout:Nothing to do. 2024-04-24T01:54:27.289 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-04-24T01:54:27.352 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install rbd-fuse 2024-04-24T01:54:27.559 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-04-24T01:54:27.616 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : rbd-mirror-2:19.0.0-3227.g424d8c43.el9.x86_64 1/1 2024-04-24T01:54:27.640 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3227.g424d8c43.el9.x86_64 1/1 2024-04-24T01:54:27.641 INFO:teuthology.orchestra.run.smithi079.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T01:54:27.641 INFO:teuthology.orchestra.run.smithi079.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-24T01:54:27.641 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-24T01:54:27.641 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-24T01:54:27.641 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:54:27.870 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:02:41 ago on Wed 24 Apr 2024 01:51:46 AM UTC. 2024-04-24T01:54:27.990 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-04-24T01:54:27.991 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-24T01:54:27.991 INFO:teuthology.orchestra.run.smithi103.stdout: Package Architecture Version Repository Size 2024-04-24T01:54:27.991 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-24T01:54:27.991 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-04-24T01:54:27.991 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-fuse x86_64 2:19.0.0-3227.g424d8c43.el9 ceph 90 k 2024-04-24T01:54:27.991 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:54:27.991 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-04-24T01:54:27.991 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-24T01:54:27.991 INFO:teuthology.orchestra.run.smithi103.stdout:Install 1 Package 2024-04-24T01:54:27.991 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:54:27.992 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 90 k 2024-04-24T01:54:27.992 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 313 k 2024-04-24T01:54:27.992 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-04-24T01:54:28.204 INFO:teuthology.orchestra.run.smithi103.stdout:rbd-fuse-19.0.0-3227.g424d8c43.el9.x86_64.rpm 422 kB/s | 90 kB 00:00 2024-04-24T01:54:28.205 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-04-24T01:54:28.205 INFO:teuthology.orchestra.run.smithi103.stdout:Total 419 kB/s | 90 kB 00:00 2024-04-24T01:54:28.206 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-04-24T01:54:28.214 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-04-24T01:54:28.214 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-04-24T01:54:28.277 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-04-24T01:54:28.277 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-04-24T01:54:28.431 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-04-24T01:54:28.533 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : rbd-fuse-2:19.0.0-3227.g424d8c43.el9.x86_64 1/1 2024-04-24T01:54:28.785 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : rbd-mirror-2:19.0.0-3227.g424d8c43.el9.x86_64 1/1 2024-04-24T01:54:28.785 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:54:28.785 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-04-24T01:54:28.785 INFO:teuthology.orchestra.run.smithi079.stdout: rbd-mirror-2:19.0.0-3227.g424d8c43.el9.x86_64 2024-04-24T01:54:28.785 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:54:28.785 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-04-24T01:54:28.889 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: rbd-fuse-2:19.0.0-3227.g424d8c43.el9.x86_64 1/1 2024-04-24T01:54:29.051 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install rbd-nbd 2024-04-24T01:54:29.116 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : rbd-fuse-2:19.0.0-3227.g424d8c43.el9.x86_64 1/1 2024-04-24T01:54:29.116 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:54:29.116 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-04-24T01:54:29.116 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-fuse-2:19.0.0-3227.g424d8c43.el9.x86_64 2024-04-24T01:54:29.116 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:54:29.116 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-04-24T01:54:29.398 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install rbd-mirror 2024-04-24T01:54:29.574 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:02:22 ago on Wed 24 Apr 2024 01:52:07 AM UTC. 2024-04-24T01:54:29.705 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-04-24T01:54:29.706 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-04-24T01:54:29.706 INFO:teuthology.orchestra.run.smithi079.stdout: Package Architecture Version Repository Size 2024-04-24T01:54:29.706 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-04-24T01:54:29.706 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-04-24T01:54:29.706 INFO:teuthology.orchestra.run.smithi079.stdout: rbd-nbd x86_64 2:19.0.0-3227.g424d8c43.el9 ceph 175 k 2024-04-24T01:54:29.706 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:54:29.706 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-04-24T01:54:29.707 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-04-24T01:54:29.707 INFO:teuthology.orchestra.run.smithi079.stdout:Install 1 Package 2024-04-24T01:54:29.707 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:54:29.707 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 175 k 2024-04-24T01:54:29.707 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 673 k 2024-04-24T01:54:29.707 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-04-24T01:54:29.906 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:02:43 ago on Wed 24 Apr 2024 01:51:46 AM UTC. 2024-04-24T01:54:30.020 INFO:teuthology.orchestra.run.smithi079.stdout:rbd-nbd-19.0.0-3227.g424d8c43.el9.x86_64.rpm 560 kB/s | 175 kB 00:00 2024-04-24T01:54:30.021 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-04-24T01:54:30.021 INFO:teuthology.orchestra.run.smithi079.stdout:Total 557 kB/s | 175 kB 00:00 2024-04-24T01:54:30.021 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-04-24T01:54:30.026 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-04-24T01:54:30.027 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-24T01:54:30.027 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repo Size 2024-04-24T01:54:30.027 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-24T01:54:30.027 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-04-24T01:54:30.027 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-mirror x86_64 2:19.0.0-3227.g424d8c43.el9 ceph 3.4 M 2024-04-24T01:54:30.028 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:54:30.028 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-04-24T01:54:30.028 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-24T01:54:30.028 INFO:teuthology.orchestra.run.smithi103.stdout:Install 1 Package 2024-04-24T01:54:30.028 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:54:30.028 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 3.4 M 2024-04-24T01:54:30.028 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 29 M 2024-04-24T01:54:30.029 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-04-24T01:54:30.031 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-04-24T01:54:30.031 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-04-24T01:54:30.095 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-04-24T01:54:30.096 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-04-24T01:54:30.252 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-04-24T01:54:30.358 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : rbd-nbd-2:19.0.0-3227.g424d8c43.el9.x86_64 1/1 2024-04-24T01:54:30.650 INFO:teuthology.orchestra.run.smithi103.stdout:rbd-mirror-19.0.0-3227.g424d8c43.el9.x86_64.rpm 5.5 MB/s | 3.4 MB 00:00 2024-04-24T01:54:30.651 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-04-24T01:54:30.651 INFO:teuthology.orchestra.run.smithi103.stdout:Total 5.5 MB/s | 3.4 MB 00:00 2024-04-24T01:54:30.651 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-04-24T01:54:30.663 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-04-24T01:54:30.663 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-04-24T01:54:30.750 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-04-24T01:54:30.750 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-04-24T01:54:30.767 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: rbd-nbd-2:19.0.0-3227.g424d8c43.el9.x86_64 1/1 2024-04-24T01:54:31.017 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : rbd-nbd-2:19.0.0-3227.g424d8c43.el9.x86_64 1/1 2024-04-24T01:54:31.017 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:54:31.017 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-04-24T01:54:31.017 INFO:teuthology.orchestra.run.smithi079.stdout: rbd-nbd-2:19.0.0-3227.g424d8c43.el9.x86_64 2024-04-24T01:54:31.017 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:54:31.017 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-04-24T01:54:31.257 DEBUG:teuthology.parallel:result is None 2024-04-24T01:54:31.352 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-04-24T01:54:31.593 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : rbd-mirror-2:19.0.0-3227.g424d8c43.el9.x86_64 1/1 2024-04-24T01:54:31.618 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3227.g424d8c43.el9.x86_64 1/1 2024-04-24T01:54:31.618 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T01:54:31.618 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-24T01:54:31.618 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-24T01:54:31.618 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-24T01:54:31.618 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:54:32.600 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : rbd-mirror-2:19.0.0-3227.g424d8c43.el9.x86_64 1/1 2024-04-24T01:54:32.600 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:54:32.600 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-04-24T01:54:32.600 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-mirror-2:19.0.0-3227.g424d8c43.el9.x86_64 2024-04-24T01:54:32.600 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:54:32.600 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-04-24T01:54:32.847 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install rbd-nbd 2024-04-24T01:54:33.363 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:02:47 ago on Wed 24 Apr 2024 01:51:46 AM UTC. 2024-04-24T01:54:33.492 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-04-24T01:54:33.493 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-24T01:54:33.493 INFO:teuthology.orchestra.run.smithi103.stdout: Package Architecture Version Repository Size 2024-04-24T01:54:33.493 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-24T01:54:33.493 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-04-24T01:54:33.493 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-nbd x86_64 2:19.0.0-3227.g424d8c43.el9 ceph 175 k 2024-04-24T01:54:33.493 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:54:33.493 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-04-24T01:54:33.493 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-04-24T01:54:33.493 INFO:teuthology.orchestra.run.smithi103.stdout:Install 1 Package 2024-04-24T01:54:33.493 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:54:33.494 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 175 k 2024-04-24T01:54:33.494 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 673 k 2024-04-24T01:54:33.494 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-04-24T01:54:33.921 INFO:teuthology.orchestra.run.smithi103.stdout:rbd-nbd-19.0.0-3227.g424d8c43.el9.x86_64.rpm 411 kB/s | 175 kB 00:00 2024-04-24T01:54:33.922 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-04-24T01:54:33.922 INFO:teuthology.orchestra.run.smithi103.stdout:Total 409 kB/s | 175 kB 00:00 2024-04-24T01:54:33.922 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-04-24T01:54:33.932 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-04-24T01:54:33.932 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-04-24T01:54:33.995 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-04-24T01:54:33.996 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-04-24T01:54:34.149 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-04-24T01:54:34.259 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : rbd-nbd-2:19.0.0-3227.g424d8c43.el9.x86_64 1/1 2024-04-24T01:54:34.680 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: rbd-nbd-2:19.0.0-3227.g424d8c43.el9.x86_64 1/1 2024-04-24T01:54:34.911 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : rbd-nbd-2:19.0.0-3227.g424d8c43.el9.x86_64 1/1 2024-04-24T01:54:34.911 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:54:34.911 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-04-24T01:54:34.912 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-nbd-2:19.0.0-3227.g424d8c43.el9.x86_64 2024-04-24T01:54:34.912 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:54:34.912 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-04-24T01:54:35.151 DEBUG:teuthology.parallel:result is None 2024-04-24T01:54:35.152 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=424d8c4389b684b0a4d72510e1b4d398d8e6c8f4 2024-04-24T01:54:35.385 DEBUG:teuthology.orchestra.run.smithi079:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-24T01:54:35.410 INFO:teuthology.orchestra.run.smithi079.stdout:19.0.0-3227.g424d8c43.el9 2024-04-24T01:54:35.411 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3227.g424d8c43.el9 2024-04-24T01:54:35.411 INFO:teuthology.task.install:The correct ceph version 19.0.0-3227.g424d8c43 is installed. 2024-04-24T01:54:35.413 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=424d8c4389b684b0a4d72510e1b4d398d8e6c8f4 2024-04-24T01:54:35.563 DEBUG:teuthology.orchestra.run.smithi103:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-24T01:54:35.585 INFO:teuthology.orchestra.run.smithi103.stdout:19.0.0-3227.g424d8c43.el9 2024-04-24T01:54:35.585 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3227.g424d8c43.el9 2024-04-24T01:54:35.585 INFO:teuthology.task.install:The correct ceph version 19.0.0-3227.g424d8c43 is installed. 2024-04-24T01:54:35.587 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-24T01:54:35.588 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-04-24T01:54:35.588 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-24T01:54:35.619 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-24T01:54:35.619 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-24T01:54:35.649 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-24T01:54:35.650 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-04-24T01:54:35.650 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/usr/bin/daemon-helper 2024-04-24T01:54:35.687 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-24T01:54:35.754 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-24T01:54:35.754 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/usr/bin/daemon-helper 2024-04-24T01:54:35.777 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-24T01:54:35.840 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-24T01:54:35.841 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-04-24T01:54:35.841 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-24T01:54:35.868 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-24T01:54:35.933 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-24T01:54:35.933 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-24T01:54:35.965 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-24T01:54:36.029 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-24T01:54:36.029 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-04-24T01:54:36.029 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/usr/bin/stdin-killer 2024-04-24T01:54:36.054 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-24T01:54:36.120 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-24T01:54:36.120 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/usr/bin/stdin-killer 2024-04-24T01:54:36.144 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-24T01:54:36.208 INFO:teuthology.run_tasks:Running task ceph... 2024-04-24T01:54:36.307 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-04-24T01:54:36.307 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod 777 /var/log/ceph 2024-04-24T01:54:36.310 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod 777 /var/log/ceph 2024-04-24T01:54:36.333 INFO:tasks.ceph:Disabling ceph logrotate... 2024-04-24T01:54:36.333 DEBUG:teuthology.orchestra.run.smithi079:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-24T01:54:36.375 DEBUG:teuthology.orchestra.run.smithi103:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-24T01:54:36.398 INFO:tasks.ceph:Creating extra log directories... 2024-04-24T01:54:36.398 DEBUG:teuthology.orchestra.run.smithi079:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-24T01:54:36.440 DEBUG:teuthology.orchestra.run.smithi103:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-24T01:54:36.473 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-04-24T01:54:36.473 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-04-24T01:54:36.473 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yingxin-2024-04-24_01:32:34-crimson-rados-ci-yingxin-crimson-load-obc-new-1-distro-default-smithi/7671138', 'branch': 'ci-yingxin-crimson-load-obc-new-1', 'description': 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/simple thrashosds-health workloads/snaps-few-objects}', 'email': 'yingxin.cheng@intel.com', 'first_in_suite': False, 'job_id': '7671138', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yingxin-2024-04-24_01:32:34-crimson-rados-ci-yingxin-crimson-load-obc-new-1-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'ci-yingxin-crimson-load-obc-new-1'}, 'ceph': {'conf': {'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum'], 'sha1': '424d8c4389b684b0a4d72510e1b4d398d8e6c8f4'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'flavor': 'crimson', 'sha1': '424d8c4389b684b0a4d72510e1b4d398d8e6c8f4'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'thrashosds': {'min_in': 4}, 'workunit': {'branch': 'ci-yingxin-crimson-load-obc-new-1', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': '424d8c4389b684b0a4d72510e1b4d398d8e6c8f4'}}, 'owner': 'scheduled_yingxin@teuthology', 'priority': 60, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'osd.0', 'osd.1', 'client.0', 'node-exporter.a'], ['mgr.x', 'osd.2', 'osd.3', 'client.1', 'prometheus.a', 'node-exporter.b']], 'sha1': '424d8c4389b684b0a4d72510e1b4d398d8e6c8f4', 'sleep_before_teardown': 0, 'suite': 'crimson-rados', 'suite_branch': 'ci-yingxin-crimson-load-obc-new-1', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_424d8c4389b684b0a4d72510e1b4d398d8e6c8f4/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '424d8c4389b684b0a4d72510e1b4d398d8e6c8f4', 'targets': {'smithi079.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBExTLpkVK6cnU8uvK6oiGFbBNE58CQpG+zqhnOeuD48hfXiTHMTNkdoxAlH5naPPwvvTytBB/PBvw2/kSuQQPGA=', 'smithi103.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJwP7wGqBu8yx2tmUKdALm5FUtcVc092/3lmeqecFmPUCzjl9i77SkaJVwmYpo5zeySSoUJukG4ObFttE6oivLk='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum'], 'sha1': '424d8c4389b684b0a4d72510e1b4d398d8e6c8f4', 'cluster': 'ceph'}}, {'thrashosds': {'ceph_objectstore_tool': False, 'chance_down': 0, 'chance_inject_pause_short': 0, 'chance_reset_purged_snaps_last': 0, 'chance_thrash_cluster_full': 0, 'chance_thrash_pg_upmap': 0, 'dump_ops_enable': False, 'min_in': 3, 'noscrub_toggle_delay': 0, 'reweight_osd': 0, 'sighup_delay': 0, 'thrash_primary_affinity': False, 'timeout': 2400}}, {'rados': {'clients': ['client.0'], 'objects': 50, 'op_weights': {'copy_from': 0, 'delete': 50, 'read': 100, 'rollback': 0, 'snap_create': 50, 'snap_remove': 50, 'write': 100}, 'ops': 4000}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17', 'timestamp': '2024-04-24_01:32:34', 'tube': 'smithi', 'user': 'yingxin', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-04-24T01:54:36.473 DEBUG:teuthology.orchestra.run.smithi079:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-24T01:54:36.504 DEBUG:teuthology.orchestra.run.smithi103:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-24T01:54:36.518 DEBUG:teuthology.orchestra.run.smithi079:> sudo install -d -m0777 -- /var/run/ceph 2024-04-24T01:54:36.559 DEBUG:teuthology.orchestra.run.smithi103:> sudo install -d -m0777 -- /var/run/ceph 2024-04-24T01:54:36.582 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-04-24T01:54:36.582 DEBUG:teuthology.orchestra.run.smithi079:> dd if=/scratch_devs of=/dev/stdout 2024-04-24T01:54:36.634 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-24T01:54:36.635 DEBUG:teuthology.orchestra.run.smithi079:> stat /dev/vg_nvme/lv_1 2024-04-24T01:54:36.689 INFO:teuthology.orchestra.run.smithi079.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-24T01:54:36.689 INFO:teuthology.orchestra.run.smithi079.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T01:54:36.689 INFO:teuthology.orchestra.run.smithi079.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-04-24T01:54:36.689 INFO:teuthology.orchestra.run.smithi079.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T01:54:36.689 INFO:teuthology.orchestra.run.smithi079.stdout:Context: system_u:object_r:device_t:s0 2024-04-24T01:54:36.689 INFO:teuthology.orchestra.run.smithi079.stdout:Access: 2024-04-24 01:54:27.881626850 +0000 2024-04-24T01:54:36.690 INFO:teuthology.orchestra.run.smithi079.stdout:Modify: 2024-04-24 01:51:07.955421167 +0000 2024-04-24T01:54:36.690 INFO:teuthology.orchestra.run.smithi079.stdout:Change: 2024-04-24 01:51:07.955421167 +0000 2024-04-24T01:54:36.690 INFO:teuthology.orchestra.run.smithi079.stdout: Birth: 2024-04-24 01:51:07.955421167 +0000 2024-04-24T01:54:36.690 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-24T01:54:36.751 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records in 2024-04-24T01:54:36.751 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records out 2024-04-24T01:54:36.751 INFO:teuthology.orchestra.run.smithi079.stderr:512 bytes copied, 0.000204389 s, 2.5 MB/s 2024-04-24T01:54:36.753 DEBUG:teuthology.orchestra.run.smithi079:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-24T01:54:36.808 DEBUG:teuthology.orchestra.run.smithi079:> stat /dev/vg_nvme/lv_2 2024-04-24T01:54:36.863 INFO:teuthology.orchestra.run.smithi079.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-24T01:54:36.864 INFO:teuthology.orchestra.run.smithi079.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T01:54:36.864 INFO:teuthology.orchestra.run.smithi079.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-04-24T01:54:36.864 INFO:teuthology.orchestra.run.smithi079.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T01:54:36.864 INFO:teuthology.orchestra.run.smithi079.stdout:Context: system_u:object_r:device_t:s0 2024-04-24T01:54:36.864 INFO:teuthology.orchestra.run.smithi079.stdout:Access: 2024-04-24 01:54:27.881626850 +0000 2024-04-24T01:54:36.864 INFO:teuthology.orchestra.run.smithi079.stdout:Modify: 2024-04-24 01:51:08.298413105 +0000 2024-04-24T01:54:36.864 INFO:teuthology.orchestra.run.smithi079.stdout:Change: 2024-04-24 01:51:08.298413105 +0000 2024-04-24T01:54:36.864 INFO:teuthology.orchestra.run.smithi079.stdout: Birth: 2024-04-24 01:51:08.298413105 +0000 2024-04-24T01:54:36.864 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-24T01:54:36.926 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records in 2024-04-24T01:54:36.926 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records out 2024-04-24T01:54:36.926 INFO:teuthology.orchestra.run.smithi079.stderr:512 bytes copied, 0.000192084 s, 2.7 MB/s 2024-04-24T01:54:36.927 DEBUG:teuthology.orchestra.run.smithi079:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-24T01:54:36.983 DEBUG:teuthology.orchestra.run.smithi079:> stat /dev/vg_nvme/lv_3 2024-04-24T01:54:37.037 INFO:teuthology.orchestra.run.smithi079.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-24T01:54:37.037 INFO:teuthology.orchestra.run.smithi079.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T01:54:37.037 INFO:teuthology.orchestra.run.smithi079.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-04-24T01:54:37.037 INFO:teuthology.orchestra.run.smithi079.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T01:54:37.037 INFO:teuthology.orchestra.run.smithi079.stdout:Context: system_u:object_r:device_t:s0 2024-04-24T01:54:37.037 INFO:teuthology.orchestra.run.smithi079.stdout:Access: 2024-04-24 01:54:27.881626850 +0000 2024-04-24T01:54:37.037 INFO:teuthology.orchestra.run.smithi079.stdout:Modify: 2024-04-24 01:51:08.623405466 +0000 2024-04-24T01:54:37.038 INFO:teuthology.orchestra.run.smithi079.stdout:Change: 2024-04-24 01:51:08.623405466 +0000 2024-04-24T01:54:37.038 INFO:teuthology.orchestra.run.smithi079.stdout: Birth: 2024-04-24 01:51:08.623405466 +0000 2024-04-24T01:54:37.038 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-24T01:54:37.099 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records in 2024-04-24T01:54:37.099 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records out 2024-04-24T01:54:37.099 INFO:teuthology.orchestra.run.smithi079.stderr:512 bytes copied, 0.000210821 s, 2.4 MB/s 2024-04-24T01:54:37.100 DEBUG:teuthology.orchestra.run.smithi079:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-24T01:54:37.156 DEBUG:teuthology.orchestra.run.smithi079:> stat /dev/vg_nvme/lv_4 2024-04-24T01:54:37.210 INFO:teuthology.orchestra.run.smithi079.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-24T01:54:37.210 INFO:teuthology.orchestra.run.smithi079.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T01:54:37.210 INFO:teuthology.orchestra.run.smithi079.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-04-24T01:54:37.211 INFO:teuthology.orchestra.run.smithi079.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T01:54:37.211 INFO:teuthology.orchestra.run.smithi079.stdout:Context: system_u:object_r:device_t:s0 2024-04-24T01:54:37.211 INFO:teuthology.orchestra.run.smithi079.stdout:Access: 2024-04-24 01:54:27.882626827 +0000 2024-04-24T01:54:37.211 INFO:teuthology.orchestra.run.smithi079.stdout:Modify: 2024-04-24 01:51:08.966397404 +0000 2024-04-24T01:54:37.211 INFO:teuthology.orchestra.run.smithi079.stdout:Change: 2024-04-24 01:51:08.966397404 +0000 2024-04-24T01:54:37.211 INFO:teuthology.orchestra.run.smithi079.stdout: Birth: 2024-04-24 01:51:08.966397404 +0000 2024-04-24T01:54:37.211 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-24T01:54:37.272 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records in 2024-04-24T01:54:37.272 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records out 2024-04-24T01:54:37.272 INFO:teuthology.orchestra.run.smithi079.stderr:512 bytes copied, 0.000220915 s, 2.3 MB/s 2024-04-24T01:54:37.274 DEBUG:teuthology.orchestra.run.smithi079:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-24T01:54:37.329 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'} 2024-04-24T01:54:37.329 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-24T01:54:37.330 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/scratch_devs of=/dev/stdout 2024-04-24T01:54:37.347 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-24T01:54:37.348 DEBUG:teuthology.orchestra.run.smithi103:> stat /dev/vg_nvme/lv_1 2024-04-24T01:54:37.404 INFO:teuthology.orchestra.run.smithi103.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-24T01:54:37.404 INFO:teuthology.orchestra.run.smithi103.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T01:54:37.404 INFO:teuthology.orchestra.run.smithi103.stdout:Device: 5h/5d Inode: 664 Links: 1 2024-04-24T01:54:37.404 INFO:teuthology.orchestra.run.smithi103.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T01:54:37.404 INFO:teuthology.orchestra.run.smithi103.stdout:Context: system_u:object_r:device_t:s0 2024-04-24T01:54:37.404 INFO:teuthology.orchestra.run.smithi103.stdout:Access: 2024-04-24 01:54:31.864916493 +0000 2024-04-24T01:54:37.405 INFO:teuthology.orchestra.run.smithi103.stdout:Modify: 2024-04-24 01:51:07.954925435 +0000 2024-04-24T01:54:37.405 INFO:teuthology.orchestra.run.smithi103.stdout:Change: 2024-04-24 01:51:07.954925435 +0000 2024-04-24T01:54:37.405 INFO:teuthology.orchestra.run.smithi103.stdout: Birth: 2024-04-24 01:51:07.954925435 +0000 2024-04-24T01:54:37.405 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-24T01:54:37.468 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records in 2024-04-24T01:54:37.469 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records out 2024-04-24T01:54:37.469 INFO:teuthology.orchestra.run.smithi103.stderr:512 bytes copied, 0.000254762 s, 2.0 MB/s 2024-04-24T01:54:37.470 DEBUG:teuthology.orchestra.run.smithi103:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-24T01:54:37.527 DEBUG:teuthology.orchestra.run.smithi103:> stat /dev/vg_nvme/lv_2 2024-04-24T01:54:37.583 INFO:teuthology.orchestra.run.smithi103.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-24T01:54:37.583 INFO:teuthology.orchestra.run.smithi103.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T01:54:37.583 INFO:teuthology.orchestra.run.smithi103.stdout:Device: 5h/5d Inode: 683 Links: 1 2024-04-24T01:54:37.584 INFO:teuthology.orchestra.run.smithi103.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T01:54:37.584 INFO:teuthology.orchestra.run.smithi103.stdout:Context: system_u:object_r:device_t:s0 2024-04-24T01:54:37.584 INFO:teuthology.orchestra.run.smithi103.stdout:Access: 2024-04-24 01:54:31.864916493 +0000 2024-04-24T01:54:37.584 INFO:teuthology.orchestra.run.smithi103.stdout:Modify: 2024-04-24 01:51:08.306918235 +0000 2024-04-24T01:54:37.584 INFO:teuthology.orchestra.run.smithi103.stdout:Change: 2024-04-24 01:51:08.306918235 +0000 2024-04-24T01:54:37.584 INFO:teuthology.orchestra.run.smithi103.stdout: Birth: 2024-04-24 01:51:08.306918235 +0000 2024-04-24T01:54:37.584 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-24T01:54:37.648 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records in 2024-04-24T01:54:37.648 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records out 2024-04-24T01:54:37.648 INFO:teuthology.orchestra.run.smithi103.stderr:512 bytes copied, 0.000286049 s, 1.8 MB/s 2024-04-24T01:54:37.649 DEBUG:teuthology.orchestra.run.smithi103:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-24T01:54:37.706 DEBUG:teuthology.orchestra.run.smithi103:> stat /dev/vg_nvme/lv_3 2024-04-24T01:54:37.762 INFO:teuthology.orchestra.run.smithi103.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-24T01:54:37.762 INFO:teuthology.orchestra.run.smithi103.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T01:54:37.762 INFO:teuthology.orchestra.run.smithi103.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-04-24T01:54:37.762 INFO:teuthology.orchestra.run.smithi103.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T01:54:37.762 INFO:teuthology.orchestra.run.smithi103.stdout:Context: system_u:object_r:device_t:s0 2024-04-24T01:54:37.762 INFO:teuthology.orchestra.run.smithi103.stdout:Access: 2024-04-24 01:54:31.865916474 +0000 2024-04-24T01:54:37.762 INFO:teuthology.orchestra.run.smithi103.stdout:Modify: 2024-04-24 01:51:08.643911341 +0000 2024-04-24T01:54:37.763 INFO:teuthology.orchestra.run.smithi103.stdout:Change: 2024-04-24 01:51:08.643911341 +0000 2024-04-24T01:54:37.763 INFO:teuthology.orchestra.run.smithi103.stdout: Birth: 2024-04-24 01:51:08.643911341 +0000 2024-04-24T01:54:37.763 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-24T01:54:37.825 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records in 2024-04-24T01:54:37.826 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records out 2024-04-24T01:54:37.826 INFO:teuthology.orchestra.run.smithi103.stderr:512 bytes copied, 0.000228967 s, 2.2 MB/s 2024-04-24T01:54:37.827 DEBUG:teuthology.orchestra.run.smithi103:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-24T01:54:37.883 DEBUG:teuthology.orchestra.run.smithi103:> stat /dev/vg_nvme/lv_4 2024-04-24T01:54:37.938 INFO:teuthology.orchestra.run.smithi103.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-24T01:54:37.938 INFO:teuthology.orchestra.run.smithi103.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T01:54:37.938 INFO:teuthology.orchestra.run.smithi103.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-04-24T01:54:37.938 INFO:teuthology.orchestra.run.smithi103.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T01:54:37.938 INFO:teuthology.orchestra.run.smithi103.stdout:Context: system_u:object_r:device_t:s0 2024-04-24T01:54:37.939 INFO:teuthology.orchestra.run.smithi103.stdout:Access: 2024-04-24 01:54:31.865916474 +0000 2024-04-24T01:54:37.939 INFO:teuthology.orchestra.run.smithi103.stdout:Modify: 2024-04-24 01:51:08.979904468 +0000 2024-04-24T01:54:37.939 INFO:teuthology.orchestra.run.smithi103.stdout:Change: 2024-04-24 01:51:08.979904468 +0000 2024-04-24T01:54:37.939 INFO:teuthology.orchestra.run.smithi103.stdout: Birth: 2024-04-24 01:51:08.979904468 +0000 2024-04-24T01:54:37.939 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-24T01:54:38.000 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records in 2024-04-24T01:54:38.000 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records out 2024-04-24T01:54:38.001 INFO:teuthology.orchestra.run.smithi103.stderr:512 bytes copied, 0.000189205 s, 2.7 MB/s 2024-04-24T01:54:38.002 DEBUG:teuthology.orchestra.run.smithi103:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-24T01:54:38.057 INFO:tasks.ceph:osd dev map: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'} 2024-04-24T01:54:38.057 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi079.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'}, Remote(name='ubuntu@smithi103.front.sepia.ceph.com'): {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'}} 2024-04-24T01:54:38.057 INFO:tasks.ceph:Generating config... 2024-04-24T01:54:38.059 INFO:tasks.ceph:[osd] debug monc = 20 2024-04-24T01:54:38.059 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-04-24T01:54:38.059 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2024-04-24T01:54:38.059 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-04-24T01:54:38.059 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-04-24T01:54:38.059 INFO:tasks.ceph:[osd] bluestore rocksdb cf = False 2024-04-24T01:54:38.059 INFO:tasks.ceph:[osd] crimson alien thread cpu cores = 6-7 2024-04-24T01:54:38.059 INFO:tasks.ceph:[osd] crimson osd obc lru size = 10 2024-04-24T01:54:38.059 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-04-24T01:54:38.059 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-04-24T01:54:38.059 INFO:tasks.ceph:[osd] debug ms = 1 2024-04-24T01:54:38.059 INFO:tasks.ceph:[osd] debug osd = 20 2024-04-24T01:54:38.059 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-04-24T01:54:38.059 INFO:tasks.ceph:[osd] err to stderr = True 2024-04-24T01:54:38.059 INFO:tasks.ceph:[osd] log flush on exit = True 2024-04-24T01:54:38.060 INFO:tasks.ceph:[osd] log to file = False 2024-04-24T01:54:38.060 INFO:tasks.ceph:[osd] log to stderr = True 2024-04-24T01:54:38.060 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-04-24T01:54:38.060 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-04-24T01:54:38.060 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-04-24T01:54:38.060 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-04-24T01:54:38.060 INFO:tasks.ceph:[osd] osd debug reject backfill probability = 0.3 2024-04-24T01:54:38.060 INFO:tasks.ceph:[osd] osd delete sleep = 1 2024-04-24T01:54:38.060 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-04-24T01:54:38.060 INFO:tasks.ceph:[osd] osd max backfills = 3 2024-04-24T01:54:38.060 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-04-24T01:54:38.060 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-04-24T01:54:38.060 INFO:tasks.ceph:[osd] osd scrub max interval = 120 2024-04-24T01:54:38.060 INFO:tasks.ceph:[osd] osd scrub min interval = 60 2024-04-24T01:54:38.060 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-04-24T01:54:38.060 INFO:tasks.ceph:[osd] osd snap trim sleep = 2 2024-04-24T01:54:38.061 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2024-04-24T01:54:38.061 INFO:tasks.ceph:[global] ms client mode = crc 2024-04-24T01:54:38.061 INFO:tasks.ceph:[global] ms cluster mode = crc 2024-04-24T01:54:38.061 INFO:tasks.ceph:[global] ms mon client mode = crc 2024-04-24T01:54:38.061 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2024-04-24T01:54:38.061 INFO:tasks.ceph:[global] ms mon service mode = crc 2024-04-24T01:54:38.061 INFO:tasks.ceph:[global] ms service mode = crc 2024-04-24T01:54:38.061 INFO:tasks.ceph:[global] osd_pool_default_min_size = 2 2024-04-24T01:54:38.061 INFO:tasks.ceph:[global] osd_pool_default_size = 3 2024-04-24T01:54:38.061 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-04-24T01:54:38.061 INFO:tasks.ceph:[mgr] debug ms = 1 2024-04-24T01:54:38.061 INFO:tasks.ceph:[mon] debug mon = 20 2024-04-24T01:54:38.061 INFO:tasks.ceph:[mon] debug ms = 1 2024-04-24T01:54:38.061 INFO:tasks.ceph:[mon] debug paxos = 20 2024-04-24T01:54:38.061 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-04-24T01:54:38.061 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-04-24T01:54:38.061 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-04-24T01:54:38.062 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-04-24T01:54:38.062 INFO:tasks.ceph:[mon] osd pool default crimson = True 2024-04-24T01:54:38.062 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-04-24T01:54:38.062 INFO:tasks.ceph:[osd.0] crimson seastar cpu cores = 0-2 2024-04-24T01:54:38.062 INFO:tasks.ceph:[osd.1] crimson seastar cpu cores = 3-5 2024-04-24T01:54:38.062 INFO:tasks.ceph:[osd.2] crimson seastar cpu cores = 0-2 2024-04-24T01:54:38.062 INFO:tasks.ceph:[osd.3] crimson seastar cpu cores = 3-5 2024-04-24T01:54:38.062 INFO:tasks.ceph:Setting up mon.a... 2024-04-24T01:54:38.062 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-04-24T01:54:38.125 INFO:teuthology.orchestra.run.smithi079.stdout:creating /etc/ceph/ceph.keyring 2024-04-24T01:54:38.128 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-04-24T01:54:38.177 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-24T01:54:38.200 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.79')] 2024-04-24T01:54:38.201 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.79', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': 120, 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}} 2024-04-24T01:54:38.202 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-04-24T01:54:38.202 DEBUG:teuthology.orchestra.run.smithi079:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-04-24T01:54:38.256 DEBUG:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.79 --print /home/ubuntu/cephtest/ceph.monmap 2024-04-24T01:54:38.344 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:54:38.343+0000 7f11fc43bf00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:54:38.345 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:54:38.344+0000 7f11fc43bf00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:54:38.345 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:54:38.344+0000 7f11fc43bf00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:54:38.345 INFO:teuthology.orchestra.run.smithi079.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-04-24T01:54:38.345 INFO:teuthology.orchestra.run.smithi079.stdout:monmaptool: generated fsid e600dd1c-0bef-4ea0-91c9-2f649f75622f 2024-04-24T01:54:38.345 INFO:teuthology.orchestra.run.smithi079.stdout:setting min_mon_release = quincy 2024-04-24T01:54:38.345 INFO:teuthology.orchestra.run.smithi079.stdout:epoch 0 2024-04-24T01:54:38.346 INFO:teuthology.orchestra.run.smithi079.stdout:fsid e600dd1c-0bef-4ea0-91c9-2f649f75622f 2024-04-24T01:54:38.346 INFO:teuthology.orchestra.run.smithi079.stdout:last_changed 2024-04-24T01:54:38.345566+0000 2024-04-24T01:54:38.346 INFO:teuthology.orchestra.run.smithi079.stdout:created 2024-04-24T01:54:38.345566+0000 2024-04-24T01:54:38.346 INFO:teuthology.orchestra.run.smithi079.stdout:min_mon_release 17 (quincy) 2024-04-24T01:54:38.346 INFO:teuthology.orchestra.run.smithi079.stdout:election_strategy: 1 2024-04-24T01:54:38.346 INFO:teuthology.orchestra.run.smithi079.stdout:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-24T01:54:38.346 INFO:teuthology.orchestra.run.smithi079.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2024-04-24T01:54:38.350 DEBUG:teuthology.orchestra.run.smithi079:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-04-24T01:54:38.365 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID e600dd1c-0bef-4ea0-91c9-2f649f75622f... 2024-04-24T01:54:38.366 DEBUG:teuthology.orchestra.run.smithi079:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-24T01:54:38.407 DEBUG:teuthology.orchestra.run.smithi103:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-24T01:54:38.444 INFO:teuthology.orchestra.run.smithi079.stdout:[global] 2024-04-24T01:54:38.444 INFO:teuthology.orchestra.run.smithi079.stdout: chdir = "" 2024-04-24T01:54:38.444 INFO:teuthology.orchestra.run.smithi079.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-24T01:54:38.444 INFO:teuthology.orchestra.run.smithi079.stdout: auth supported = cephx 2024-04-24T01:54:38.444 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:54:38.444 INFO:teuthology.orchestra.run.smithi079.stdout: filestore xattr use omap = true 2024-04-24T01:54:38.444 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:54:38.444 INFO:teuthology.orchestra.run.smithi079.stdout: mon clock drift allowed = 1.000 2024-04-24T01:54:38.445 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:54:38.445 INFO:teuthology.orchestra.run.smithi079.stdout: osd crush chooseleaf type = 0 2024-04-24T01:54:38.445 INFO:teuthology.orchestra.run.smithi079.stdout: auth debug = true 2024-04-24T01:54:38.445 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:54:38.445 INFO:teuthology.orchestra.run.smithi079.stdout: ms die on old message = true 2024-04-24T01:54:38.445 INFO:teuthology.orchestra.run.smithi079.stdout: ms die on bug = true 2024-04-24T01:54:38.445 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:54:38.445 INFO:teuthology.orchestra.run.smithi079.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-24T01:54:38.445 INFO:teuthology.orchestra.run.smithi079.stdout: mon pg warn max object skew = 0 2024-04-24T01:54:38.445 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:54:38.445 INFO:teuthology.orchestra.run.smithi079.stdout: # disable pg_autoscaler by default for new pools 2024-04-24T01:54:38.445 INFO:teuthology.orchestra.run.smithi079.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-24T01:54:38.445 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:54:38.445 INFO:teuthology.orchestra.run.smithi079.stdout: osd pool default size = 2 2024-04-24T01:54:38.446 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:54:38.446 INFO:teuthology.orchestra.run.smithi079.stdout: mon osd allow primary affinity = true 2024-04-24T01:54:38.446 INFO:teuthology.orchestra.run.smithi079.stdout: mon osd allow pg remap = true 2024-04-24T01:54:38.446 INFO:teuthology.orchestra.run.smithi079.stdout: mon warn on legacy crush tunables = false 2024-04-24T01:54:38.446 INFO:teuthology.orchestra.run.smithi079.stdout: mon warn on crush straw calc version zero = false 2024-04-24T01:54:38.446 INFO:teuthology.orchestra.run.smithi079.stdout: mon warn on no sortbitwise = false 2024-04-24T01:54:38.446 INFO:teuthology.orchestra.run.smithi079.stdout: mon warn on osd down out interval zero = false 2024-04-24T01:54:38.446 INFO:teuthology.orchestra.run.smithi079.stdout: mon warn on too few osds = false 2024-04-24T01:54:38.446 INFO:teuthology.orchestra.run.smithi079.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-24T01:54:38.446 INFO:teuthology.orchestra.run.smithi079.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-24T01:54:38.446 INFO:teuthology.orchestra.run.smithi079.stdout: mon_allow_pool_size_one = true 2024-04-24T01:54:38.446 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:54:38.446 INFO:teuthology.orchestra.run.smithi079.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-24T01:54:38.447 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:54:38.447 INFO:teuthology.orchestra.run.smithi079.stdout: osd default data pool replay window = 5 2024-04-24T01:54:38.447 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:54:38.447 INFO:teuthology.orchestra.run.smithi079.stdout: mon allow pool delete = true 2024-04-24T01:54:38.447 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:54:38.447 INFO:teuthology.orchestra.run.smithi079.stdout: mon cluster log file level = debug 2024-04-24T01:54:38.447 INFO:teuthology.orchestra.run.smithi079.stdout: debug asserts on shutdown = true 2024-04-24T01:54:38.447 INFO:teuthology.orchestra.run.smithi079.stdout: mon health detail to clog = false 2024-04-24T01:54:38.447 INFO:teuthology.orchestra.run.smithi079.stdout: mon host = 172.21.15.79 2024-04-24T01:54:38.447 INFO:teuthology.orchestra.run.smithi079.stdout: enable experimental unrecoverable data corrupting features = crimson 2024-04-24T01:54:38.447 INFO:teuthology.orchestra.run.smithi079.stdout: ms client mode = crc 2024-04-24T01:54:38.447 INFO:teuthology.orchestra.run.smithi079.stdout: ms cluster mode = crc 2024-04-24T01:54:38.447 INFO:teuthology.orchestra.run.smithi079.stdout: ms mon client mode = crc 2024-04-24T01:54:38.448 INFO:teuthology.orchestra.run.smithi079.stdout: ms mon cluster mode = crc 2024-04-24T01:54:38.448 INFO:teuthology.orchestra.run.smithi079.stdout: ms mon service mode = crc 2024-04-24T01:54:38.448 INFO:teuthology.orchestra.run.smithi079.stdout: ms service mode = crc 2024-04-24T01:54:38.448 INFO:teuthology.orchestra.run.smithi079.stdout: osd_pool_default_min_size = 2 2024-04-24T01:54:38.448 INFO:teuthology.orchestra.run.smithi079.stdout: osd_pool_default_size = 3 2024-04-24T01:54:38.448 INFO:teuthology.orchestra.run.smithi079.stdout: fsid = e600dd1c-0bef-4ea0-91c9-2f649f75622f 2024-04-24T01:54:38.448 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:54:38.448 INFO:teuthology.orchestra.run.smithi079.stdout:[osd] 2024-04-24T01:54:38.449 INFO:teuthology.orchestra.run.smithi079.stdout: osd journal size = 100 2024-04-24T01:54:38.449 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:54:38.449 INFO:teuthology.orchestra.run.smithi079.stdout: osd scrub load threshold = 5.0 2024-04-24T01:54:38.449 INFO:teuthology.orchestra.run.smithi079.stdout: osd scrub max interval = 120 2024-04-24T01:54:38.449 INFO:teuthology.orchestra.run.smithi079.stdout: osd mclock profile = high_recovery_ops 2024-04-24T01:54:38.449 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:54:38.449 INFO:teuthology.orchestra.run.smithi079.stdout: osd recover clone overlap = true 2024-04-24T01:54:38.449 INFO:teuthology.orchestra.run.smithi079.stdout: osd recovery max chunk = 1048576 2024-04-24T01:54:38.449 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:54:38.449 INFO:teuthology.orchestra.run.smithi079.stdout: osd debug shutdown = true 2024-04-24T01:54:38.449 INFO:teuthology.orchestra.run.smithi079.stdout: osd debug op order = true 2024-04-24T01:54:38.449 INFO:teuthology.orchestra.run.smithi079.stdout: osd debug verify stray on activate = true 2024-04-24T01:54:38.449 INFO:teuthology.orchestra.run.smithi079.stdout: osd debug trim objects = true 2024-04-24T01:54:38.449 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:54:38.450 INFO:teuthology.orchestra.run.smithi079.stdout: osd open classes on start = true 2024-04-24T01:54:38.450 INFO:teuthology.orchestra.run.smithi079.stdout: osd debug pg log writeout = true 2024-04-24T01:54:38.450 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:54:38.450 INFO:teuthology.orchestra.run.smithi079.stdout: osd deep scrub update digest min age = 30 2024-04-24T01:54:38.450 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:54:38.450 INFO:teuthology.orchestra.run.smithi079.stdout: osd map max advance = 10 2024-04-24T01:54:38.450 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:54:38.450 INFO:teuthology.orchestra.run.smithi079.stdout: journal zero on create = true 2024-04-24T01:54:38.450 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:54:38.450 INFO:teuthology.orchestra.run.smithi079.stdout: filestore ondisk finisher threads = 3 2024-04-24T01:54:38.450 INFO:teuthology.orchestra.run.smithi079.stdout: filestore apply finisher threads = 3 2024-04-24T01:54:38.450 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:54:38.450 INFO:teuthology.orchestra.run.smithi079.stdout: bdev debug aio = true 2024-04-24T01:54:38.450 INFO:teuthology.orchestra.run.smithi079.stdout: osd debug misdirected ops = true 2024-04-24T01:54:38.451 INFO:teuthology.orchestra.run.smithi079.stdout: debug monc = 20 2024-04-24T01:54:38.451 INFO:teuthology.orchestra.run.smithi079.stdout: bluestore block size = 96636764160 2024-04-24T01:54:38.451 INFO:teuthology.orchestra.run.smithi079.stdout: bluestore compression algorithm = snappy 2024-04-24T01:54:38.451 INFO:teuthology.orchestra.run.smithi079.stdout: bluestore compression mode = aggressive 2024-04-24T01:54:38.451 INFO:teuthology.orchestra.run.smithi079.stdout: bluestore fsck on mount = True 2024-04-24T01:54:38.451 INFO:teuthology.orchestra.run.smithi079.stdout: bluestore rocksdb cf = False 2024-04-24T01:54:38.451 INFO:teuthology.orchestra.run.smithi079.stdout: crimson alien thread cpu cores = 6-7 2024-04-24T01:54:38.451 INFO:teuthology.orchestra.run.smithi079.stdout: crimson osd obc lru size = 10 2024-04-24T01:54:38.451 INFO:teuthology.orchestra.run.smithi079.stdout: debug bluefs = 20 2024-04-24T01:54:38.451 INFO:teuthology.orchestra.run.smithi079.stdout: debug bluestore = 20 2024-04-24T01:54:38.451 INFO:teuthology.orchestra.run.smithi079.stdout: debug ms = 1 2024-04-24T01:54:38.451 INFO:teuthology.orchestra.run.smithi079.stdout: debug osd = 20 2024-04-24T01:54:38.451 INFO:teuthology.orchestra.run.smithi079.stdout: debug rocksdb = 10 2024-04-24T01:54:38.451 INFO:teuthology.orchestra.run.smithi079.stdout: err to stderr = True 2024-04-24T01:54:38.451 INFO:teuthology.orchestra.run.smithi079.stdout: log flush on exit = True 2024-04-24T01:54:38.452 INFO:teuthology.orchestra.run.smithi079.stdout: log to file = False 2024-04-24T01:54:38.452 INFO:teuthology.orchestra.run.smithi079.stdout: log to stderr = True 2024-04-24T01:54:38.452 INFO:teuthology.orchestra.run.smithi079.stdout: mon osd backfillfull_ratio = 0.85 2024-04-24T01:54:38.452 INFO:teuthology.orchestra.run.smithi079.stdout: mon osd full ratio = 0.9 2024-04-24T01:54:38.452 INFO:teuthology.orchestra.run.smithi079.stdout: mon osd nearfull ratio = 0.8 2024-04-24T01:54:38.452 INFO:teuthology.orchestra.run.smithi079.stdout: osd blocked scrub grace period = 3600 2024-04-24T01:54:38.452 INFO:teuthology.orchestra.run.smithi079.stdout: osd debug reject backfill probability = 0.3 2024-04-24T01:54:38.452 INFO:teuthology.orchestra.run.smithi079.stdout: osd delete sleep = 1 2024-04-24T01:54:38.452 INFO:teuthology.orchestra.run.smithi079.stdout: osd failsafe full ratio = 0.95 2024-04-24T01:54:38.452 INFO:teuthology.orchestra.run.smithi079.stdout: osd max backfills = 3 2024-04-24T01:54:38.452 INFO:teuthology.orchestra.run.smithi079.stdout: osd max markdown count = 1000 2024-04-24T01:54:38.452 INFO:teuthology.orchestra.run.smithi079.stdout: osd objectstore = bluestore 2024-04-24T01:54:38.452 INFO:teuthology.orchestra.run.smithi079.stdout: osd scrub min interval = 60 2024-04-24T01:54:38.452 INFO:teuthology.orchestra.run.smithi079.stdout: osd shutdown pgref assert = True 2024-04-24T01:54:38.452 INFO:teuthology.orchestra.run.smithi079.stdout: osd snap trim sleep = 2 2024-04-24T01:54:38.453 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:54:38.453 INFO:teuthology.orchestra.run.smithi079.stdout:[mgr] 2024-04-24T01:54:38.453 INFO:teuthology.orchestra.run.smithi079.stdout: debug ms = 1 2024-04-24T01:54:38.453 INFO:teuthology.orchestra.run.smithi079.stdout: debug mgr = 20 2024-04-24T01:54:38.453 INFO:teuthology.orchestra.run.smithi079.stdout: debug mon = 20 2024-04-24T01:54:38.453 INFO:teuthology.orchestra.run.smithi079.stdout: debug auth = 20 2024-04-24T01:54:38.453 INFO:teuthology.orchestra.run.smithi079.stdout: mon reweight min pgs per osd = 4 2024-04-24T01:54:38.453 INFO:teuthology.orchestra.run.smithi079.stdout: mon reweight min bytes per osd = 10 2024-04-24T01:54:38.453 INFO:teuthology.orchestra.run.smithi079.stdout: mgr/telemetry/nag = false 2024-04-24T01:54:38.453 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:54:38.453 INFO:teuthology.orchestra.run.smithi079.stdout:[mon] 2024-04-24T01:54:38.453 INFO:teuthology.orchestra.run.smithi079.stdout: debug ms = 1 2024-04-24T01:54:38.453 INFO:teuthology.orchestra.run.smithi079.stdout: debug mon = 20 2024-04-24T01:54:38.453 INFO:teuthology.orchestra.run.smithi079.stdout: debug paxos = 20 2024-04-24T01:54:38.454 INFO:teuthology.orchestra.run.smithi079.stdout: debug auth = 20 2024-04-24T01:54:38.454 INFO:teuthology.orchestra.run.smithi079.stdout: mon data avail warn = 5 2024-04-24T01:54:38.454 INFO:teuthology.orchestra.run.smithi079.stdout: mon mgr mkfs grace = 240 2024-04-24T01:54:38.454 INFO:teuthology.orchestra.run.smithi079.stdout: mon reweight min pgs per osd = 4 2024-04-24T01:54:38.454 INFO:teuthology.orchestra.run.smithi079.stdout: mon osd reporter subtree level = osd 2024-04-24T01:54:38.454 INFO:teuthology.orchestra.run.smithi079.stdout: mon osd prime pg temp = true 2024-04-24T01:54:38.454 INFO:teuthology.orchestra.run.smithi079.stdout: mon reweight min bytes per osd = 10 2024-04-24T01:54:38.454 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:54:38.454 INFO:teuthology.orchestra.run.smithi079.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-24T01:54:38.454 INFO:teuthology.orchestra.run.smithi079.stdout: auth mon ticket ttl = 660 # 11m 2024-04-24T01:54:38.454 INFO:teuthology.orchestra.run.smithi079.stdout: auth service ticket ttl = 240 # 4m 2024-04-24T01:54:38.454 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:54:38.454 INFO:teuthology.orchestra.run.smithi079.stdout: # don't complain about insecure global_id in the test suite 2024-04-24T01:54:38.455 INFO:teuthology.orchestra.run.smithi079.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-24T01:54:38.455 INFO:teuthology.orchestra.run.smithi079.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-24T01:54:38.455 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:54:38.455 INFO:teuthology.orchestra.run.smithi079.stdout: # 1m isn't quite enough 2024-04-24T01:54:38.455 INFO:teuthology.orchestra.run.smithi079.stdout: mon_down_mkfs_grace = 2m 2024-04-24T01:54:38.455 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:54:38.455 INFO:teuthology.orchestra.run.smithi079.stdout: mon_warn_on_filestore_osds = false 2024-04-24T01:54:38.455 INFO:teuthology.orchestra.run.smithi079.stdout: mon min osdmap epochs = 50 2024-04-24T01:54:38.455 INFO:teuthology.orchestra.run.smithi079.stdout: mon osdmap full prune interval = 2 2024-04-24T01:54:38.455 INFO:teuthology.orchestra.run.smithi079.stdout: mon osdmap full prune min = 15 2024-04-24T01:54:38.455 INFO:teuthology.orchestra.run.smithi079.stdout: mon osdmap full prune txsize = 2 2024-04-24T01:54:38.455 INFO:teuthology.orchestra.run.smithi079.stdout: osd pool default crimson = True 2024-04-24T01:54:38.455 INFO:teuthology.orchestra.run.smithi079.stdout: paxos service trim min = 10 2024-04-24T01:54:38.455 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:54:38.455 INFO:teuthology.orchestra.run.smithi079.stdout:[client] 2024-04-24T01:54:38.456 INFO:teuthology.orchestra.run.smithi079.stdout: rgw cache enabled = true 2024-04-24T01:54:38.456 INFO:teuthology.orchestra.run.smithi079.stdout: rgw enable ops log = true 2024-04-24T01:54:38.456 INFO:teuthology.orchestra.run.smithi079.stdout: rgw enable usage log = true 2024-04-24T01:54:38.456 INFO:teuthology.orchestra.run.smithi079.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-24T01:54:38.456 INFO:teuthology.orchestra.run.smithi079.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-24T01:54:38.456 INFO:teuthology.orchestra.run.smithi079.stdout:[mon.a] 2024-04-24T01:54:38.456 INFO:teuthology.orchestra.run.smithi079.stdout:[osd.0] 2024-04-24T01:54:38.456 INFO:teuthology.orchestra.run.smithi079.stdout: crimson seastar cpu cores = 0-2 2024-04-24T01:54:38.456 INFO:teuthology.orchestra.run.smithi079.stdout:[osd.1] 2024-04-24T01:54:38.456 INFO:teuthology.orchestra.run.smithi079.stdout: crimson seastar cpu cores = 3-5 2024-04-24T01:54:38.456 INFO:teuthology.orchestra.run.smithi079.stdout:[osd.2] 2024-04-24T01:54:38.456 INFO:teuthology.orchestra.run.smithi079.stdout: crimson seastar cpu cores = 0-2 2024-04-24T01:54:38.456 INFO:teuthology.orchestra.run.smithi079.stdout:[osd.3] 2024-04-24T01:54:38.456 INFO:teuthology.orchestra.run.smithi079.stdout: crimson seastar cpu cores = 3-5 2024-04-24T01:54:38.457 INFO:teuthology.orchestra.run.smithi103.stdout:[global] 2024-04-24T01:54:38.457 INFO:teuthology.orchestra.run.smithi103.stdout: chdir = "" 2024-04-24T01:54:38.457 INFO:teuthology.orchestra.run.smithi103.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-24T01:54:38.457 INFO:teuthology.orchestra.run.smithi103.stdout: auth supported = cephx 2024-04-24T01:54:38.457 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:54:38.457 INFO:teuthology.orchestra.run.smithi103.stdout: filestore xattr use omap = true 2024-04-24T01:54:38.457 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:54:38.458 INFO:teuthology.orchestra.run.smithi103.stdout: mon clock drift allowed = 1.000 2024-04-24T01:54:38.458 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:54:38.458 INFO:teuthology.orchestra.run.smithi103.stdout: osd crush chooseleaf type = 0 2024-04-24T01:54:38.458 INFO:teuthology.orchestra.run.smithi103.stdout: auth debug = true 2024-04-24T01:54:38.458 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:54:38.458 INFO:teuthology.orchestra.run.smithi103.stdout: ms die on old message = true 2024-04-24T01:54:38.458 INFO:teuthology.orchestra.run.smithi103.stdout: ms die on bug = true 2024-04-24T01:54:38.458 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:54:38.458 INFO:teuthology.orchestra.run.smithi103.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-24T01:54:38.458 INFO:teuthology.orchestra.run.smithi103.stdout: mon pg warn max object skew = 0 2024-04-24T01:54:38.458 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:54:38.458 INFO:teuthology.orchestra.run.smithi103.stdout: # disable pg_autoscaler by default for new pools 2024-04-24T01:54:38.458 INFO:teuthology.orchestra.run.smithi103.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-24T01:54:38.458 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:54:38.459 INFO:teuthology.orchestra.run.smithi103.stdout: osd pool default size = 2 2024-04-24T01:54:38.459 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:54:38.459 INFO:teuthology.orchestra.run.smithi103.stdout: mon osd allow primary affinity = true 2024-04-24T01:54:38.459 INFO:teuthology.orchestra.run.smithi103.stdout: mon osd allow pg remap = true 2024-04-24T01:54:38.459 INFO:teuthology.orchestra.run.smithi103.stdout: mon warn on legacy crush tunables = false 2024-04-24T01:54:38.459 INFO:teuthology.orchestra.run.smithi103.stdout: mon warn on crush straw calc version zero = false 2024-04-24T01:54:38.459 INFO:teuthology.orchestra.run.smithi103.stdout: mon warn on no sortbitwise = false 2024-04-24T01:54:38.459 INFO:teuthology.orchestra.run.smithi103.stdout: mon warn on osd down out interval zero = false 2024-04-24T01:54:38.459 INFO:teuthology.orchestra.run.smithi103.stdout: mon warn on too few osds = false 2024-04-24T01:54:38.459 INFO:teuthology.orchestra.run.smithi103.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-24T01:54:38.459 INFO:teuthology.orchestra.run.smithi103.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-24T01:54:38.460 INFO:teuthology.orchestra.run.smithi103.stdout: mon_allow_pool_size_one = true 2024-04-24T01:54:38.460 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:54:38.460 INFO:teuthology.orchestra.run.smithi103.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-24T01:54:38.460 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:54:38.460 INFO:teuthology.orchestra.run.smithi103.stdout: osd default data pool replay window = 5 2024-04-24T01:54:38.460 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:54:38.460 INFO:teuthology.orchestra.run.smithi103.stdout: mon allow pool delete = true 2024-04-24T01:54:38.460 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:54:38.460 INFO:teuthology.orchestra.run.smithi103.stdout: mon cluster log file level = debug 2024-04-24T01:54:38.461 INFO:teuthology.orchestra.run.smithi103.stdout: debug asserts on shutdown = true 2024-04-24T01:54:38.461 INFO:teuthology.orchestra.run.smithi103.stdout: mon health detail to clog = false 2024-04-24T01:54:38.461 INFO:teuthology.orchestra.run.smithi103.stdout: mon host = 172.21.15.79 2024-04-24T01:54:38.461 INFO:teuthology.orchestra.run.smithi103.stdout: enable experimental unrecoverable data corrupting features = crimson 2024-04-24T01:54:38.461 INFO:teuthology.orchestra.run.smithi103.stdout: ms client mode = crc 2024-04-24T01:54:38.461 INFO:teuthology.orchestra.run.smithi103.stdout: ms cluster mode = crc 2024-04-24T01:54:38.461 INFO:teuthology.orchestra.run.smithi103.stdout: ms mon client mode = crc 2024-04-24T01:54:38.462 INFO:teuthology.orchestra.run.smithi103.stdout: ms mon cluster mode = crc 2024-04-24T01:54:38.462 INFO:teuthology.orchestra.run.smithi103.stdout: ms mon service mode = crc 2024-04-24T01:54:38.462 INFO:teuthology.orchestra.run.smithi103.stdout: ms service mode = crc 2024-04-24T01:54:38.462 INFO:teuthology.orchestra.run.smithi103.stdout: osd_pool_default_min_size = 2 2024-04-24T01:54:38.462 INFO:teuthology.orchestra.run.smithi103.stdout: osd_pool_default_size = 3 2024-04-24T01:54:38.462 INFO:teuthology.orchestra.run.smithi103.stdout: fsid = e600dd1c-0bef-4ea0-91c9-2f649f75622f 2024-04-24T01:54:38.462 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:54:38.463 INFO:teuthology.orchestra.run.smithi103.stdout:[osd] 2024-04-24T01:54:38.463 INFO:teuthology.orchestra.run.smithi103.stdout: osd journal size = 100 2024-04-24T01:54:38.463 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:54:38.463 INFO:teuthology.orchestra.run.smithi103.stdout: osd scrub load threshold = 5.0 2024-04-24T01:54:38.463 INFO:teuthology.orchestra.run.smithi103.stdout: osd scrub max interval = 120 2024-04-24T01:54:38.463 INFO:teuthology.orchestra.run.smithi103.stdout: osd mclock profile = high_recovery_ops 2024-04-24T01:54:38.463 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:54:38.464 INFO:teuthology.orchestra.run.smithi103.stdout: osd recover clone overlap = true 2024-04-24T01:54:38.464 INFO:teuthology.orchestra.run.smithi103.stdout: osd recovery max chunk = 1048576 2024-04-24T01:54:38.464 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:54:38.464 INFO:teuthology.orchestra.run.smithi103.stdout: osd debug shutdown = true 2024-04-24T01:54:38.464 INFO:teuthology.orchestra.run.smithi103.stdout: osd debug op order = true 2024-04-24T01:54:38.464 INFO:teuthology.orchestra.run.smithi103.stdout: osd debug verify stray on activate = true 2024-04-24T01:54:38.464 INFO:teuthology.orchestra.run.smithi103.stdout: osd debug trim objects = true 2024-04-24T01:54:38.465 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:54:38.465 INFO:teuthology.orchestra.run.smithi103.stdout: osd open classes on start = true 2024-04-24T01:54:38.465 INFO:teuthology.orchestra.run.smithi103.stdout: osd debug pg log writeout = true 2024-04-24T01:54:38.465 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:54:38.465 INFO:teuthology.orchestra.run.smithi103.stdout: osd deep scrub update digest min age = 30 2024-04-24T01:54:38.465 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:54:38.465 INFO:teuthology.orchestra.run.smithi103.stdout: osd map max advance = 10 2024-04-24T01:54:38.466 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:54:38.466 INFO:teuthology.orchestra.run.smithi103.stdout: journal zero on create = true 2024-04-24T01:54:38.466 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:54:38.466 INFO:teuthology.orchestra.run.smithi103.stdout: filestore ondisk finisher threads = 3 2024-04-24T01:54:38.466 INFO:teuthology.orchestra.run.smithi103.stdout: filestore apply finisher threads = 3 2024-04-24T01:54:38.466 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:54:38.467 INFO:teuthology.orchestra.run.smithi103.stdout: bdev debug aio = true 2024-04-24T01:54:38.467 INFO:teuthology.orchestra.run.smithi103.stdout: osd debug misdirected ops = true 2024-04-24T01:54:38.467 INFO:teuthology.orchestra.run.smithi103.stdout: debug monc = 20 2024-04-24T01:54:38.467 INFO:teuthology.orchestra.run.smithi103.stdout: bluestore block size = 96636764160 2024-04-24T01:54:38.467 INFO:teuthology.orchestra.run.smithi103.stdout: bluestore compression algorithm = snappy 2024-04-24T01:54:38.467 INFO:teuthology.orchestra.run.smithi103.stdout: bluestore compression mode = aggressive 2024-04-24T01:54:38.467 INFO:teuthology.orchestra.run.smithi103.stdout: bluestore fsck on mount = True 2024-04-24T01:54:38.468 INFO:teuthology.orchestra.run.smithi103.stdout: bluestore rocksdb cf = False 2024-04-24T01:54:38.468 INFO:teuthology.orchestra.run.smithi103.stdout: crimson alien thread cpu cores = 6-7 2024-04-24T01:54:38.468 INFO:teuthology.orchestra.run.smithi103.stdout: crimson osd obc lru size = 10 2024-04-24T01:54:38.468 INFO:teuthology.orchestra.run.smithi103.stdout: debug bluefs = 20 2024-04-24T01:54:38.468 INFO:teuthology.orchestra.run.smithi103.stdout: debug bluestore = 20 2024-04-24T01:54:38.468 INFO:teuthology.orchestra.run.smithi103.stdout: debug ms = 1 2024-04-24T01:54:38.469 INFO:teuthology.orchestra.run.smithi103.stdout: debug osd = 20 2024-04-24T01:54:38.469 INFO:teuthology.orchestra.run.smithi103.stdout: debug rocksdb = 10 2024-04-24T01:54:38.469 INFO:teuthology.orchestra.run.smithi103.stdout: err to stderr = True 2024-04-24T01:54:38.469 INFO:teuthology.orchestra.run.smithi103.stdout: log flush on exit = True 2024-04-24T01:54:38.469 INFO:teuthology.orchestra.run.smithi103.stdout: log to file = False 2024-04-24T01:54:38.469 INFO:teuthology.orchestra.run.smithi103.stdout: log to stderr = True 2024-04-24T01:54:38.469 INFO:teuthology.orchestra.run.smithi103.stdout: mon osd backfillfull_ratio = 0.85 2024-04-24T01:54:38.469 INFO:teuthology.orchestra.run.smithi103.stdout: mon osd full ratio = 0.9 2024-04-24T01:54:38.470 INFO:teuthology.orchestra.run.smithi103.stdout: mon osd nearfull ratio = 0.8 2024-04-24T01:54:38.470 INFO:teuthology.orchestra.run.smithi103.stdout: osd blocked scrub grace period = 3600 2024-04-24T01:54:38.470 INFO:teuthology.orchestra.run.smithi103.stdout: osd debug reject backfill probability = 0.3 2024-04-24T01:54:38.470 INFO:teuthology.orchestra.run.smithi103.stdout: osd delete sleep = 1 2024-04-24T01:54:38.470 INFO:teuthology.orchestra.run.smithi103.stdout: osd failsafe full ratio = 0.95 2024-04-24T01:54:38.470 INFO:teuthology.orchestra.run.smithi103.stdout: osd max backfills = 3 2024-04-24T01:54:38.470 INFO:teuthology.orchestra.run.smithi103.stdout: osd max markdown count = 1000 2024-04-24T01:54:38.471 INFO:teuthology.orchestra.run.smithi103.stdout: osd objectstore = bluestore 2024-04-24T01:54:38.471 INFO:teuthology.orchestra.run.smithi103.stdout: osd scrub min interval = 60 2024-04-24T01:54:38.471 INFO:teuthology.orchestra.run.smithi103.stdout: osd shutdown pgref assert = True 2024-04-24T01:54:38.471 INFO:teuthology.orchestra.run.smithi103.stdout: osd snap trim sleep = 2 2024-04-24T01:54:38.471 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:54:38.471 INFO:teuthology.orchestra.run.smithi103.stdout:[mgr] 2024-04-24T01:54:38.472 INFO:teuthology.orchestra.run.smithi103.stdout: debug ms = 1 2024-04-24T01:54:38.472 INFO:teuthology.orchestra.run.smithi103.stdout: debug mgr = 20 2024-04-24T01:54:38.472 INFO:teuthology.orchestra.run.smithi103.stdout: debug mon = 20 2024-04-24T01:54:38.472 INFO:teuthology.orchestra.run.smithi103.stdout: debug auth = 20 2024-04-24T01:54:38.472 INFO:teuthology.orchestra.run.smithi103.stdout: mon reweight min pgs per osd = 4 2024-04-24T01:54:38.472 INFO:teuthology.orchestra.run.smithi103.stdout: mon reweight min bytes per osd = 10 2024-04-24T01:54:38.472 INFO:teuthology.orchestra.run.smithi103.stdout: mgr/telemetry/nag = false 2024-04-24T01:54:38.473 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:54:38.473 INFO:teuthology.orchestra.run.smithi103.stdout:[mon] 2024-04-24T01:54:38.473 INFO:teuthology.orchestra.run.smithi103.stdout: debug ms = 1 2024-04-24T01:54:38.473 INFO:teuthology.orchestra.run.smithi103.stdout: debug mon = 20 2024-04-24T01:54:38.473 INFO:teuthology.orchestra.run.smithi103.stdout: debug paxos = 20 2024-04-24T01:54:38.473 INFO:teuthology.orchestra.run.smithi103.stdout: debug auth = 20 2024-04-24T01:54:38.474 INFO:teuthology.orchestra.run.smithi103.stdout: mon data avail warn = 5 2024-04-24T01:54:38.474 INFO:teuthology.orchestra.run.smithi103.stdout: mon mgr mkfs grace = 240 2024-04-24T01:54:38.474 INFO:teuthology.orchestra.run.smithi103.stdout: mon reweight min pgs per osd = 4 2024-04-24T01:54:38.474 INFO:teuthology.orchestra.run.smithi103.stdout: mon osd reporter subtree level = osd 2024-04-24T01:54:38.474 INFO:teuthology.orchestra.run.smithi103.stdout: mon osd prime pg temp = true 2024-04-24T01:54:38.474 INFO:teuthology.orchestra.run.smithi103.stdout: mon reweight min bytes per osd = 10 2024-04-24T01:54:38.475 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:54:38.475 INFO:teuthology.orchestra.run.smithi103.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-24T01:54:38.475 INFO:teuthology.orchestra.run.smithi103.stdout: auth mon ticket ttl = 660 # 11m 2024-04-24T01:54:38.475 INFO:teuthology.orchestra.run.smithi103.stdout: auth service ticket ttl = 240 # 4m 2024-04-24T01:54:38.475 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:54:38.475 INFO:teuthology.orchestra.run.smithi103.stdout: # don't complain about insecure global_id in the test suite 2024-04-24T01:54:38.475 INFO:teuthology.orchestra.run.smithi103.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-24T01:54:38.476 INFO:teuthology.orchestra.run.smithi103.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-24T01:54:38.476 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:54:38.476 INFO:teuthology.orchestra.run.smithi103.stdout: # 1m isn't quite enough 2024-04-24T01:54:38.476 INFO:teuthology.orchestra.run.smithi103.stdout: mon_down_mkfs_grace = 2m 2024-04-24T01:54:38.476 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:54:38.476 INFO:teuthology.orchestra.run.smithi103.stdout: mon_warn_on_filestore_osds = false 2024-04-24T01:54:38.477 INFO:teuthology.orchestra.run.smithi103.stdout: mon min osdmap epochs = 50 2024-04-24T01:54:38.477 INFO:teuthology.orchestra.run.smithi103.stdout: mon osdmap full prune interval = 2 2024-04-24T01:54:38.477 INFO:teuthology.orchestra.run.smithi103.stdout: mon osdmap full prune min = 15 2024-04-24T01:54:38.477 INFO:teuthology.orchestra.run.smithi103.stdout: mon osdmap full prune txsize = 2 2024-04-24T01:54:38.477 INFO:teuthology.orchestra.run.smithi103.stdout: osd pool default crimson = True 2024-04-24T01:54:38.477 INFO:teuthology.orchestra.run.smithi103.stdout: paxos service trim min = 10 2024-04-24T01:54:38.477 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T01:54:38.478 INFO:teuthology.orchestra.run.smithi103.stdout:[client] 2024-04-24T01:54:38.478 INFO:teuthology.orchestra.run.smithi103.stdout: rgw cache enabled = true 2024-04-24T01:54:38.478 INFO:teuthology.orchestra.run.smithi103.stdout: rgw enable ops log = true 2024-04-24T01:54:38.478 INFO:teuthology.orchestra.run.smithi103.stdout: rgw enable usage log = true 2024-04-24T01:54:38.478 INFO:teuthology.orchestra.run.smithi103.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-24T01:54:38.478 INFO:teuthology.orchestra.run.smithi103.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-24T01:54:38.479 INFO:teuthology.orchestra.run.smithi103.stdout:[mon.a] 2024-04-24T01:54:38.479 INFO:teuthology.orchestra.run.smithi103.stdout:[osd.0] 2024-04-24T01:54:38.479 INFO:teuthology.orchestra.run.smithi103.stdout: crimson seastar cpu cores = 0-2 2024-04-24T01:54:38.479 INFO:teuthology.orchestra.run.smithi103.stdout:[osd.1] 2024-04-24T01:54:38.479 INFO:teuthology.orchestra.run.smithi103.stdout: crimson seastar cpu cores = 3-5 2024-04-24T01:54:38.479 INFO:teuthology.orchestra.run.smithi103.stdout:[osd.2] 2024-04-24T01:54:38.479 INFO:teuthology.orchestra.run.smithi103.stdout: crimson seastar cpu cores = 0-2 2024-04-24T01:54:38.480 INFO:teuthology.orchestra.run.smithi103.stdout:[osd.3] 2024-04-24T01:54:38.480 INFO:teuthology.orchestra.run.smithi103.stdout: crimson seastar cpu cores = 3-5 2024-04-24T01:54:38.482 INFO:tasks.ceph:Creating admin key on mon.a... 2024-04-24T01:54:38.482 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-04-24T01:54:38.584 INFO:tasks.ceph:Copying monmap to all nodes... 2024-04-24T01:54:38.584 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-04-24T01:54:38.584 DEBUG:teuthology.orchestra.run.smithi079:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-04-24T01:54:38.599 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-04-24T01:54:38.599 DEBUG:teuthology.orchestra.run.smithi079:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-04-24T01:54:38.653 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi079.front.sepia.ceph.com 2024-04-24T01:54:38.653 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-04-24T01:54:38.653 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-24T01:54:38.653 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-24T01:54:38.726 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-04-24T01:54:38.726 DEBUG:teuthology.orchestra.run.smithi079:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-24T01:54:38.781 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi103.front.sepia.ceph.com 2024-04-24T01:54:38.781 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-24T01:54:38.781 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-24T01:54:38.782 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-24T01:54:38.815 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-24T01:54:38.815 DEBUG:teuthology.orchestra.run.smithi103:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-24T01:54:38.870 INFO:tasks.ceph:Setting up mon nodes... 2024-04-24T01:54:38.870 INFO:tasks.ceph:Setting up mgr nodes... 2024-04-24T01:54:38.870 DEBUG:teuthology.orchestra.run.smithi103:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-04-24T01:54:38.976 INFO:teuthology.orchestra.run.smithi103.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-04-24T01:54:38.979 INFO:tasks.ceph:Setting up mds nodes... 2024-04-24T01:54:38.979 INFO:tasks.ceph_client:Setting up client nodes... 2024-04-24T01:54:38.979 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-24T01:54:39.027 INFO:teuthology.orchestra.run.smithi079.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-04-24T01:54:39.039 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-04-24T01:54:39.087 INFO:teuthology.orchestra.run.smithi103.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-04-24T01:54:39.100 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-04-24T01:54:39.100 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi079.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'}, Remote(name='ubuntu@smithi103.front.sepia.ceph.com'): {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'}} 2024-04-24T01:54:39.100 DEBUG:teuthology.orchestra.run.smithi079:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-04-24T01:54:39.123 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'} 2024-04-24T01:54:39.124 INFO:tasks.ceph:role: osd.0 2024-04-24T01:54:39.124 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi079.front.sepia.ceph.com 2024-04-24T01:54:39.124 DEBUG:teuthology.orchestra.run.smithi079:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-24T01:54:39.194 INFO:teuthology.orchestra.run.smithi079.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-24T01:54:39.194 INFO:teuthology.orchestra.run.smithi079.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-24T01:54:39.194 INFO:teuthology.orchestra.run.smithi079.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-24T01:54:39.194 INFO:teuthology.orchestra.run.smithi079.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-24T01:54:39.194 INFO:teuthology.orchestra.run.smithi079.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-24T01:54:39.194 INFO:teuthology.orchestra.run.smithi079.stdout: = sunit=0 swidth=0 blks 2024-04-24T01:54:39.194 INFO:teuthology.orchestra.run.smithi079.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-24T01:54:39.194 INFO:teuthology.orchestra.run.smithi079.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-24T01:54:39.194 INFO:teuthology.orchestra.run.smithi079.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-24T01:54:39.194 INFO:teuthology.orchestra.run.smithi079.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-24T01:54:39.707 INFO:teuthology.orchestra.run.smithi079.stdout:Discarding blocks...Done. 2024-04-24T01:54:39.709 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi079.front.sepia.ceph.com -o noatime 2024-04-24T01:54:39.709 DEBUG:teuthology.orchestra.run.smithi079:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-04-24T01:54:39.833 DEBUG:teuthology.orchestra.run.smithi079:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-04-24T01:54:39.863 DEBUG:teuthology.orchestra.run.smithi079:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-04-24T01:54:39.927 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'} 2024-04-24T01:54:39.928 INFO:tasks.ceph:role: osd.1 2024-04-24T01:54:39.928 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi079.front.sepia.ceph.com 2024-04-24T01:54:39.928 DEBUG:teuthology.orchestra.run.smithi079:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-24T01:54:39.998 INFO:teuthology.orchestra.run.smithi079.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-24T01:54:39.998 INFO:teuthology.orchestra.run.smithi079.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-24T01:54:39.998 INFO:teuthology.orchestra.run.smithi079.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-24T01:54:39.998 INFO:teuthology.orchestra.run.smithi079.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-24T01:54:39.999 INFO:teuthology.orchestra.run.smithi079.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-24T01:54:39.999 INFO:teuthology.orchestra.run.smithi079.stdout: = sunit=0 swidth=0 blks 2024-04-24T01:54:39.999 INFO:teuthology.orchestra.run.smithi079.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-24T01:54:39.999 INFO:teuthology.orchestra.run.smithi079.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-24T01:54:39.999 INFO:teuthology.orchestra.run.smithi079.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-24T01:54:39.999 INFO:teuthology.orchestra.run.smithi079.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-24T01:54:40.456 INFO:teuthology.orchestra.run.smithi079.stdout:Discarding blocks...Done. 2024-04-24T01:54:40.458 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi079.front.sepia.ceph.com -o noatime 2024-04-24T01:54:40.458 DEBUG:teuthology.orchestra.run.smithi079:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-04-24T01:54:40.548 DEBUG:teuthology.orchestra.run.smithi079:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-04-24T01:54:40.576 DEBUG:teuthology.orchestra.run.smithi079:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-24T01:54:40.866 INFO:teuthology.orchestra.run.smithi079.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-24T01:54:40.866 INFO:teuthology.orchestra.run.smithi079.stderr:INFO 2024-04-24 01:54:40,867 seastar - Reactor backend: linux-aio 2024-04-24T01:54:40.877 INFO:teuthology.orchestra.run.smithi079.stderr:==26604==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-24T01:54:40.989 INFO:teuthology.orchestra.run.smithi079.stderr:INFO 2024-04-24 01:54:40,989 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-04-24T01:54:40.989 INFO:teuthology.orchestra.run.smithi079.stderr:INFO 2024-04-24 01:54:40,989 [shard 0:main] osd - get_early_config: set --cpuset 0-2 2024-04-24T01:54:40.990 INFO:teuthology.orchestra.run.smithi079.stderr:DEBUG 2024-04-24 01:54:40,990 [shard 0:main] seastar - reactor::drain 2024-04-24T01:54:41.513 INFO:teuthology.orchestra.run.smithi079.stderr:----------------------------------------------------- 2024-04-24T01:54:41.513 INFO:teuthology.orchestra.run.smithi079.stderr:Suppressions used: 2024-04-24T01:54:41.513 INFO:teuthology.orchestra.run.smithi079.stderr: count bytes template 2024-04-24T01:54:41.514 INFO:teuthology.orchestra.run.smithi079.stderr: 1 8 MallocExtension::Register 2024-04-24T01:54:41.514 INFO:teuthology.orchestra.run.smithi079.stderr:----------------------------------------------------- 2024-04-24T01:54:41.514 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-24T01:54:41.568 INFO:teuthology.orchestra.run.smithi079.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-24T01:54:41.572 INFO:teuthology.orchestra.run.smithi079.stderr:INFO 2024-04-24 01:54:41,572 seastar - Reactor backend: linux-aio 2024-04-24T01:54:41.580 INFO:teuthology.orchestra.run.smithi079.stderr:==26601==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-24T01:54:46.197 INFO:teuthology.orchestra.run.smithi079.stderr:----------------------------------------------------- 2024-04-24T01:54:46.197 INFO:teuthology.orchestra.run.smithi079.stderr:Suppressions used: 2024-04-24T01:54:46.198 INFO:teuthology.orchestra.run.smithi079.stderr: count bytes template 2024-04-24T01:54:46.198 INFO:teuthology.orchestra.run.smithi079.stderr: 1 8 MallocExtension::Register 2024-04-24T01:54:46.198 INFO:teuthology.orchestra.run.smithi079.stderr:----------------------------------------------------- 2024-04-24T01:54:46.198 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-24T01:54:46.198 INFO:teuthology.orchestra.run.smithi079.stdout:created object store /var/lib/ceph/osd/ceph-0 for osd.0 fsid e600dd1c-0bef-4ea0-91c9-2f649f75622f 2024-04-24T01:54:46.220 DEBUG:teuthology.orchestra.run.smithi079:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-24T01:54:46.244 DEBUG:teuthology.orchestra.run.smithi079:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-24T01:54:46.523 INFO:teuthology.orchestra.run.smithi079.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-24T01:54:46.524 INFO:teuthology.orchestra.run.smithi079.stderr:INFO 2024-04-24 01:54:46,525 seastar - Reactor backend: linux-aio 2024-04-24T01:54:46.529 INFO:teuthology.orchestra.run.smithi079.stderr:==26786==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-24T01:54:46.633 INFO:teuthology.orchestra.run.smithi079.stderr:Reactor stalled for 65 ms on shard 0. Backtrace: 0x6bddd 0xb9754f9 0xb847fc0 0xb84953c 0xb849752 0xb8498a8 0xb849d71 0x3e6ef 0xc9f70 0xca7a6 0x3409b78 0x342726f 0x34274c4 0x5d441fb 0x5d661c4 0x5d6645e 0x5d4c846 0x5b9119d 0x5bac261 0x6e84bd4 0x6e963a7 0x6e97844 0x6e978c9 0x6e979cb 0x6e97ac3 0x6e97cd0 0x6e97eb6 0x6e98205 0x6e985a9 0x6e987d9 0x6e98a43 0x6e98d10 0x6e98f2e 0x6e98fe6 0x6e990ac 0x6e9918d 0x59f9a96 0x59f9ba0 0x59f9c5c 0x59f9d18 0x59f9dd4 0x59f9f21 0x59fa0ac 0x59fa46d 0x5a163ae 0xb85ac45 0xb874edf 0xb915edd 0xb91752b 0xb5f3d11 0xb5f468d 0x376291a 0x3763473 0x3602d7a 0x2958f 0x2963f 0x33ef444 2024-04-24T01:54:46.634 INFO:teuthology.orchestra.run.smithi079.stderr:kernel callstack: 2024-04-24T01:54:46.642 INFO:teuthology.orchestra.run.smithi079.stderr:INFO 2024-04-24 01:54:46,643 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-04-24T01:54:46.643 INFO:teuthology.orchestra.run.smithi079.stderr:INFO 2024-04-24 01:54:46,643 [shard 0:main] osd - get_early_config: set --cpuset 3-5 2024-04-24T01:54:46.643 INFO:teuthology.orchestra.run.smithi079.stderr:DEBUG 2024-04-24 01:54:46,644 [shard 0:main] seastar - reactor::drain 2024-04-24T01:54:47.175 INFO:teuthology.orchestra.run.smithi079.stderr:----------------------------------------------------- 2024-04-24T01:54:47.175 INFO:teuthology.orchestra.run.smithi079.stderr:Suppressions used: 2024-04-24T01:54:47.175 INFO:teuthology.orchestra.run.smithi079.stderr: count bytes template 2024-04-24T01:54:47.175 INFO:teuthology.orchestra.run.smithi079.stderr: 1 8 MallocExtension::Register 2024-04-24T01:54:47.175 INFO:teuthology.orchestra.run.smithi079.stderr:----------------------------------------------------- 2024-04-24T01:54:47.175 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-24T01:54:47.227 INFO:teuthology.orchestra.run.smithi079.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-24T01:54:47.231 INFO:teuthology.orchestra.run.smithi079.stderr:INFO 2024-04-24 01:54:47,231 seastar - Reactor backend: linux-aio 2024-04-24T01:54:47.246 INFO:teuthology.orchestra.run.smithi079.stderr:==26784==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-24T01:54:51.879 INFO:teuthology.orchestra.run.smithi079.stderr:----------------------------------------------------- 2024-04-24T01:54:51.879 INFO:teuthology.orchestra.run.smithi079.stderr:Suppressions used: 2024-04-24T01:54:51.879 INFO:teuthology.orchestra.run.smithi079.stderr: count bytes template 2024-04-24T01:54:51.879 INFO:teuthology.orchestra.run.smithi079.stderr: 1 8 MallocExtension::Register 2024-04-24T01:54:51.879 INFO:teuthology.orchestra.run.smithi079.stderr:----------------------------------------------------- 2024-04-24T01:54:51.879 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-24T01:54:51.879 INFO:teuthology.orchestra.run.smithi079.stdout:created object store /var/lib/ceph/osd/ceph-1 for osd.1 fsid e600dd1c-0bef-4ea0-91c9-2f649f75622f 2024-04-24T01:54:51.901 DEBUG:teuthology.orchestra.run.smithi079:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-24T01:54:51.924 DEBUG:teuthology.orchestra.run.smithi103:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-04-24T01:54:51.953 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'} 2024-04-24T01:54:51.953 INFO:tasks.ceph:role: osd.2 2024-04-24T01:54:51.953 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi103.front.sepia.ceph.com 2024-04-24T01:54:51.953 DEBUG:teuthology.orchestra.run.smithi103:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-24T01:54:52.028 INFO:teuthology.orchestra.run.smithi103.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-24T01:54:52.028 INFO:teuthology.orchestra.run.smithi103.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-24T01:54:52.028 INFO:teuthology.orchestra.run.smithi103.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-24T01:54:52.028 INFO:teuthology.orchestra.run.smithi103.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-24T01:54:52.028 INFO:teuthology.orchestra.run.smithi103.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-24T01:54:52.028 INFO:teuthology.orchestra.run.smithi103.stdout: = sunit=0 swidth=0 blks 2024-04-24T01:54:52.028 INFO:teuthology.orchestra.run.smithi103.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-24T01:54:52.028 INFO:teuthology.orchestra.run.smithi103.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-24T01:54:52.028 INFO:teuthology.orchestra.run.smithi103.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-24T01:54:52.029 INFO:teuthology.orchestra.run.smithi103.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-24T01:54:52.546 INFO:teuthology.orchestra.run.smithi103.stdout:Discarding blocks...Done. 2024-04-24T01:54:52.549 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi103.front.sepia.ceph.com -o noatime 2024-04-24T01:54:52.549 DEBUG:teuthology.orchestra.run.smithi103:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-2 2024-04-24T01:54:52.701 DEBUG:teuthology.orchestra.run.smithi103:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-04-24T01:54:52.735 DEBUG:teuthology.orchestra.run.smithi103:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-04-24T01:54:52.803 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'} 2024-04-24T01:54:52.803 INFO:tasks.ceph:role: osd.3 2024-04-24T01:54:52.803 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi103.front.sepia.ceph.com 2024-04-24T01:54:52.803 DEBUG:teuthology.orchestra.run.smithi103:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-24T01:54:52.875 INFO:teuthology.orchestra.run.smithi103.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-24T01:54:52.876 INFO:teuthology.orchestra.run.smithi103.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-24T01:54:52.876 INFO:teuthology.orchestra.run.smithi103.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-24T01:54:52.876 INFO:teuthology.orchestra.run.smithi103.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-24T01:54:52.876 INFO:teuthology.orchestra.run.smithi103.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-24T01:54:52.876 INFO:teuthology.orchestra.run.smithi103.stdout: = sunit=0 swidth=0 blks 2024-04-24T01:54:52.876 INFO:teuthology.orchestra.run.smithi103.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-24T01:54:52.876 INFO:teuthology.orchestra.run.smithi103.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-24T01:54:52.876 INFO:teuthology.orchestra.run.smithi103.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-24T01:54:52.876 INFO:teuthology.orchestra.run.smithi103.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-24T01:54:53.318 INFO:teuthology.orchestra.run.smithi103.stdout:Discarding blocks...Done. 2024-04-24T01:54:53.320 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi103.front.sepia.ceph.com -o noatime 2024-04-24T01:54:53.320 DEBUG:teuthology.orchestra.run.smithi103:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-3 2024-04-24T01:54:53.405 DEBUG:teuthology.orchestra.run.smithi103:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-04-24T01:54:53.433 DEBUG:teuthology.orchestra.run.smithi103:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-24T01:54:53.730 INFO:teuthology.orchestra.run.smithi103.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-24T01:54:53.731 INFO:teuthology.orchestra.run.smithi103.stderr:INFO 2024-04-24 01:54:53,732 seastar - Reactor backend: linux-aio 2024-04-24T01:54:53.746 INFO:teuthology.orchestra.run.smithi103.stderr:==25961==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-24T01:54:53.863 INFO:teuthology.orchestra.run.smithi103.stderr:INFO 2024-04-24 01:54:53,863 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-04-24T01:54:53.863 INFO:teuthology.orchestra.run.smithi103.stderr:INFO 2024-04-24 01:54:53,863 [shard 0:main] osd - get_early_config: set --cpuset 0-2 2024-04-24T01:54:53.863 INFO:teuthology.orchestra.run.smithi103.stderr:DEBUG 2024-04-24 01:54:53,864 [shard 0:main] seastar - reactor::drain 2024-04-24T01:54:54.419 INFO:teuthology.orchestra.run.smithi103.stderr:----------------------------------------------------- 2024-04-24T01:54:54.419 INFO:teuthology.orchestra.run.smithi103.stderr:Suppressions used: 2024-04-24T01:54:54.420 INFO:teuthology.orchestra.run.smithi103.stderr: count bytes template 2024-04-24T01:54:54.420 INFO:teuthology.orchestra.run.smithi103.stderr: 1 8 MallocExtension::Register 2024-04-24T01:54:54.420 INFO:teuthology.orchestra.run.smithi103.stderr:----------------------------------------------------- 2024-04-24T01:54:54.420 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-04-24T01:54:54.478 INFO:teuthology.orchestra.run.smithi103.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-24T01:54:54.482 INFO:teuthology.orchestra.run.smithi103.stderr:INFO 2024-04-24 01:54:54,482 seastar - Reactor backend: linux-aio 2024-04-24T01:54:54.490 INFO:teuthology.orchestra.run.smithi103.stderr:==25959==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-24T01:54:59.127 INFO:teuthology.orchestra.run.smithi103.stderr:----------------------------------------------------- 2024-04-24T01:54:59.127 INFO:teuthology.orchestra.run.smithi103.stderr:Suppressions used: 2024-04-24T01:54:59.127 INFO:teuthology.orchestra.run.smithi103.stderr: count bytes template 2024-04-24T01:54:59.127 INFO:teuthology.orchestra.run.smithi103.stderr: 1 8 MallocExtension::Register 2024-04-24T01:54:59.127 INFO:teuthology.orchestra.run.smithi103.stderr:----------------------------------------------------- 2024-04-24T01:54:59.128 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-04-24T01:54:59.128 INFO:teuthology.orchestra.run.smithi103.stdout:created object store /var/lib/ceph/osd/ceph-2 for osd.2 fsid e600dd1c-0bef-4ea0-91c9-2f649f75622f 2024-04-24T01:54:59.150 DEBUG:teuthology.orchestra.run.smithi103:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-24T01:54:59.174 DEBUG:teuthology.orchestra.run.smithi103:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-24T01:54:59.461 INFO:teuthology.orchestra.run.smithi103.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-24T01:54:59.462 INFO:teuthology.orchestra.run.smithi103.stderr:INFO 2024-04-24 01:54:59,462 seastar - Reactor backend: linux-aio 2024-04-24T01:54:59.467 INFO:teuthology.orchestra.run.smithi103.stderr:==26144==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-24T01:54:59.581 INFO:teuthology.orchestra.run.smithi103.stderr:INFO 2024-04-24 01:54:59,581 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-04-24T01:54:59.581 INFO:teuthology.orchestra.run.smithi103.stderr:INFO 2024-04-24 01:54:59,581 [shard 0:main] osd - get_early_config: set --cpuset 3-5 2024-04-24T01:54:59.581 INFO:teuthology.orchestra.run.smithi103.stderr:DEBUG 2024-04-24 01:54:59,582 [shard 0:main] seastar - reactor::drain 2024-04-24T01:55:00.112 INFO:teuthology.orchestra.run.smithi103.stderr:----------------------------------------------------- 2024-04-24T01:55:00.112 INFO:teuthology.orchestra.run.smithi103.stderr:Suppressions used: 2024-04-24T01:55:00.112 INFO:teuthology.orchestra.run.smithi103.stderr: count bytes template 2024-04-24T01:55:00.112 INFO:teuthology.orchestra.run.smithi103.stderr: 1 8 MallocExtension::Register 2024-04-24T01:55:00.113 INFO:teuthology.orchestra.run.smithi103.stderr:----------------------------------------------------- 2024-04-24T01:55:00.113 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-04-24T01:55:00.169 INFO:teuthology.orchestra.run.smithi103.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-24T01:55:00.172 INFO:teuthology.orchestra.run.smithi103.stderr:INFO 2024-04-24 01:55:00,173 seastar - Reactor backend: linux-aio 2024-04-24T01:55:00.186 INFO:teuthology.orchestra.run.smithi103.stderr:==26142==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-24T01:55:04.813 INFO:teuthology.orchestra.run.smithi103.stderr:----------------------------------------------------- 2024-04-24T01:55:04.814 INFO:teuthology.orchestra.run.smithi103.stderr:Suppressions used: 2024-04-24T01:55:04.814 INFO:teuthology.orchestra.run.smithi103.stderr: count bytes template 2024-04-24T01:55:04.814 INFO:teuthology.orchestra.run.smithi103.stderr: 1 8 MallocExtension::Register 2024-04-24T01:55:04.814 INFO:teuthology.orchestra.run.smithi103.stderr:----------------------------------------------------- 2024-04-24T01:55:04.814 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-04-24T01:55:04.814 INFO:teuthology.orchestra.run.smithi103.stdout:created object store /var/lib/ceph/osd/ceph-3 for osd.3 fsid e600dd1c-0bef-4ea0-91c9-2f649f75622f 2024-04-24T01:55:04.836 DEBUG:teuthology.orchestra.run.smithi103:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-24T01:55:04.860 INFO:tasks.ceph:Reading keys from all nodes... 2024-04-24T01:55:04.860 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-04-24T01:55:04.860 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-04-24T01:55:04.889 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-04-24T01:55:04.889 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-04-24T01:55:04.956 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-24T01:55:04.956 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-04-24T01:55:04.980 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-24T01:55:04.980 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-04-24T01:55:05.044 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-24T01:55:05.044 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-04-24T01:55:05.108 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-04-24T01:55:05.108 DEBUG:teuthology.orchestra.run.smithi079:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-04-24T01:55:05.125 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-24T01:55:05.126 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-04-24T01:55:05.163 INFO:tasks.ceph:Adding keys to all mons... 2024-04-24T01:55:05.164 DEBUG:teuthology.orchestra.run.smithi079:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-24T01:55:05.190 INFO:teuthology.orchestra.run.smithi079.stdout:[osd.0] 2024-04-24T01:55:05.190 INFO:teuthology.orchestra.run.smithi079.stdout: key = AQBhZihm2ViqKhAAa3R7qg3aQJH53nQPn+aLRw== 2024-04-24T01:55:05.190 INFO:teuthology.orchestra.run.smithi079.stdout:[osd.1] 2024-04-24T01:55:05.191 INFO:teuthology.orchestra.run.smithi079.stdout: key = AQBnZihmABS4FhAA8tm42DIlLmsxtFK0sti3Pg== 2024-04-24T01:55:05.191 INFO:teuthology.orchestra.run.smithi079.stdout:[mgr.x] 2024-04-24T01:55:05.191 INFO:teuthology.orchestra.run.smithi079.stdout: key = AQBeZihmc988OhAAYJXjOtSnpHicVr3aToceHw== 2024-04-24T01:55:05.191 INFO:teuthology.orchestra.run.smithi079.stdout:[osd.2] 2024-04-24T01:55:05.191 INFO:teuthology.orchestra.run.smithi079.stdout: key = AQBuZihmUkNyJRAA1KsoeQMJPaOx/kCz+48+8g== 2024-04-24T01:55:05.191 INFO:teuthology.orchestra.run.smithi079.stdout:[osd.3] 2024-04-24T01:55:05.191 INFO:teuthology.orchestra.run.smithi079.stdout: key = AQB0ZihmRBI2ExAAVwYq/pVuTjdnoM8Sp4QRuQ== 2024-04-24T01:55:05.191 INFO:teuthology.orchestra.run.smithi079.stdout:[client.0] 2024-04-24T01:55:05.191 INFO:teuthology.orchestra.run.smithi079.stdout: key = AQBfZihmsMmnARAAjn9NIjci9hVl6Y4EYap8Sw== 2024-04-24T01:55:05.191 INFO:teuthology.orchestra.run.smithi079.stdout:[client.1] 2024-04-24T01:55:05.191 INFO:teuthology.orchestra.run.smithi079.stdout: key = AQBfZihmQZM9BRAApgVSu9pK3q5P4Bno4hN08A== 2024-04-24T01:55:05.192 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T01:55:05.295 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T01:55:05.356 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-24T01:55:05.414 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T01:55:05.472 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T01:55:05.530 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-24T01:55:05.594 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-24T01:55:05.655 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-04-24T01:55:05.656 DEBUG:teuthology.orchestra.run.smithi079:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-04-24T01:55:05.679 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-24T01:55:05.778 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:05.777+0000 7f444b90ea80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:05.778 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:05.777+0000 7f444b90ea80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:05.791 DEBUG:teuthology.orchestra.run.smithi079:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-04-24T01:55:05.815 DEBUG:teuthology.orchestra.run.smithi079:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-24T01:55:05.869 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-04-24T01:55:05.869 INFO:tasks.ceph.mon.a:Restarting daemon 2024-04-24T01:55:05.870 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-04-24T01:55:05.912 INFO:tasks.ceph.mon.a:Started 2024-04-24T01:55:05.912 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-04-24T01:55:05.912 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-04-24T01:55:05.913 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-04-24T01:55:05.915 INFO:tasks.ceph.mgr.x:Started 2024-04-24T01:55:05.915 INFO:tasks.ceph:Setting crush tunables to default 2024-04-24T01:55:05.915 DEBUG:teuthology.orchestra.run.smithi079:> sudo ceph --cluster ceph osd crush tunables default 2024-04-24T01:55:05.991 INFO:tasks.ceph.mon.a.smithi079.stderr:2024-04-24T01:55:05.989+0000 7fb8587d5a80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:05.991 INFO:tasks.ceph.mon.a.smithi079.stderr:2024-04-24T01:55:05.989+0000 7fb8587d5a80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:05.991 INFO:tasks.ceph.mon.a.smithi079.stderr:2024-04-24T01:55:05.990+0000 7fb8587d5a80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:06.012 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-24T01:55:06.011+0000 7f0022d6d000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:06.022 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:06.015+0000 7f6b6659d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:06.030 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:06.029+0000 7f6b6659d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:06.031 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-24T01:55:06.029+0000 7f0022d6d000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:06.031 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-24T01:55:06.030+0000 7f0022d6d000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:06.101 INFO:teuthology.orchestra.run.smithi079.stderr:adjusted tunables profile to default 2024-04-24T01:55:06.112 INFO:tasks.ceph:check_enable_crimson: True 2024-04-24T01:55:06.113 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2024-04-24T01:55:06.113 DEBUG:teuthology.orchestra.run.smithi079:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2024-04-24T01:55:06.134 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-24T01:55:06.133+0000 7f0022d6d000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-24T01:55:06.217 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:06.211+0000 7f653939d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:06.221 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:06.220+0000 7f653939d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:06.280 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-24T01:55:06.280+0000 7f0022d6d000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-24T01:55:06.301 INFO:tasks.ceph.mon.a.smithi079.stderr:2024-04-24T01:55:06.300+0000 7fb8505f4640 -1 WARNING: experimental feature 'crimson' is enabled 2024-04-24T01:55:06.301 INFO:tasks.ceph.mon.a.smithi079.stderr:Please be aware that this feature is experimental, untested, 2024-04-24T01:55:06.301 INFO:tasks.ceph.mon.a.smithi079.stderr:unsupported, and may result in data corruption, data loss, 2024-04-24T01:55:06.301 INFO:tasks.ceph.mon.a.smithi079.stderr:and/or irreparable damage to your cluster. Do not use 2024-04-24T01:55:06.301 INFO:tasks.ceph.mon.a.smithi079.stderr:feature with important data. 2024-04-24T01:55:06.301 INFO:tasks.ceph.mon.a.smithi079.stderr: 2024-04-24T01:55:06.333 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-24T01:55:06.332+0000 7f0022d6d000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-24T01:55:06.583 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-24T01:55:06.582+0000 7f0022d6d000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-24T01:55:06.627 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-24T01:55:06.627+0000 7f0022d6d000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-24T01:55:07.115 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-04-24T01:55:07.115 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-04-24T01:55:07.115 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-04-24T01:55:07.139 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-04-24T01:55:07.139 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-04-24T01:55:07.204 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-24T01:55:07.204 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-04-24T01:55:07.228 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-24T01:55:07.228 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-04-24T01:55:07.292 DEBUG:teuthology.orchestra.run.smithi103:> sudo ceph --cluster ceph osd new 1b20709f-f1e7-4d59-8402-0d13620f747e 0 2024-04-24T01:55:07.308 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-24T01:55:07.307+0000 7f0022d6d000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-24T01:55:07.401 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-24T01:55:07.400+0000 7f0022d6d000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-24T01:55:07.413 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-24T01:55:07.406+0000 7f800bf9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:07.416 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-24T01:55:07.415+0000 7f800bf9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:07.446 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-24T01:55:07.446+0000 7f0022d6d000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-24T01:55:07.494 INFO:teuthology.orchestra.run.smithi103.stdout:0 2024-04-24T01:55:07.505 DEBUG:teuthology.orchestra.run.smithi103:> sudo ceph --cluster ceph osd new 54fd0eee-6d68-4c92-9ac7-6b21c87783f6 1 2024-04-24T01:55:07.611 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-24T01:55:07.605+0000 7f597235d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:07.614 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-24T01:55:07.614+0000 7f597235d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:07.621 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-24T01:55:07.620+0000 7f0022d6d000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-24T01:55:07.669 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-24T01:55:07.669+0000 7f0022d6d000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-24T01:55:07.692 INFO:teuthology.orchestra.run.smithi103.stdout:1 2024-04-24T01:55:07.703 DEBUG:teuthology.orchestra.run.smithi103:> sudo ceph --cluster ceph osd new 35820c96-5f2a-4e2e-bc4b-714ab8e1c841 2 2024-04-24T01:55:07.725 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-24T01:55:07.725+0000 7f0022d6d000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-24T01:55:07.808 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-24T01:55:07.802+0000 7fd72b9e7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:07.811 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-24T01:55:07.811+0000 7fd72b9e7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:07.845 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-24T01:55:07.844+0000 7f0022d6d000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-24T01:55:07.889 INFO:teuthology.orchestra.run.smithi103.stdout:2 2024-04-24T01:55:07.900 DEBUG:teuthology.orchestra.run.smithi103:> sudo ceph --cluster ceph osd new da975ea1-eeb6-4619-b73f-786b46dd8fe6 3 2024-04-24T01:55:08.045 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-24T01:55:08.039+0000 7f5295f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:08.048 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-24T01:55:08.048+0000 7f5295f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:08.129 INFO:teuthology.orchestra.run.smithi103.stdout:3 2024-04-24T01:55:08.140 DEBUG:teuthology.orchestra.run.smithi079:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2024-04-24T01:55:08.173 INFO:tasks.ceph.osd.0:Restarting daemon 2024-04-24T01:55:08.173 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2024-04-24T01:55:08.216 INFO:tasks.ceph.osd.0:Started 2024-04-24T01:55:08.216 DEBUG:teuthology.orchestra.run.smithi079:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2024-04-24T01:55:08.234 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-24T01:55:08.233+0000 7f0022d6d000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-24T01:55:08.246 INFO:tasks.ceph.osd.1:Restarting daemon 2024-04-24T01:55:08.247 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2024-04-24T01:55:08.289 INFO:tasks.ceph.osd.1:Started 2024-04-24T01:55:08.289 DEBUG:teuthology.orchestra.run.smithi103:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2024-04-24T01:55:08.318 INFO:tasks.ceph.osd.2:Restarting daemon 2024-04-24T01:55:08.318 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2024-04-24T01:55:08.361 INFO:tasks.ceph.osd.2:Started 2024-04-24T01:55:08.361 DEBUG:teuthology.orchestra.run.smithi103:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.3.log 2024-04-24T01:55:08.388 INFO:tasks.ceph.osd.3:Restarting daemon 2024-04-24T01:55:08.388 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2>> /var/log/ceph/ceph-osd.3.log 2024-04-24T01:55:08.431 INFO:tasks.ceph.osd.3:Started 2024-04-24T01:55:08.431 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T01:55:08.510 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-24T01:55:08.509+0000 7f0022d6d000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-24T01:55:08.544 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:08.539+0000 7faa6db5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:08.548 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:08.547+0000 7faa6db5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:08.570 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-24T01:55:08.569+0000 7f0022d6d000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-24T01:55:08.617 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:55:08.617 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":7,"fsid":"e600dd1c-0bef-4ea0-91c9-2f649f75622f","created":"2024-04-24T01:55:06.017534+0000","modified":"2024-04-24T01:55:08.126530+0000","last_up_change":"0.000000","last_in_change":"2024-04-24T01:55:08.126530+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"1b20709f-f1e7-4d59-8402-0d13620f747e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"54fd0eee-6d68-4c92-9ac7-6b21c87783f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"35820c96-5f2a-4e2e-bc4b-714ab8e1c841","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"da975ea1-eeb6-4619-b73f-786b46dd8fe6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T01:55:08.629 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-04-24T01:55:08.629 INFO:tasks.ceph:Waiting for OSDs to come up 2024-04-24T01:55:08.709 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-24T01:55:08.708+0000 7f0022d6d000 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-24T01:55:09.033 DEBUG:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-24T01:55:09.137 INFO:teuthology.misc.health.smithi079.stderr:2024-04-24T01:55:09.132+0000 7f553a99d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:09.141 INFO:teuthology.misc.health.smithi079.stderr:2024-04-24T01:55:09.140+0000 7f553a99d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:09.205 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-24T01:55:09.204+0000 7f0022d6d000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-24T01:55:09.210 INFO:teuthology.misc.health.smithi079.stdout: 2024-04-24T01:55:09.210 INFO:teuthology.misc.health.smithi079.stdout:{"epoch":7,"fsid":"e600dd1c-0bef-4ea0-91c9-2f649f75622f","created":"2024-04-24T01:55:06.017534+0000","modified":"2024-04-24T01:55:08.126530+0000","last_up_change":"0.000000","last_in_change":"2024-04-24T01:55:08.126530+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"1b20709f-f1e7-4d59-8402-0d13620f747e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"54fd0eee-6d68-4c92-9ac7-6b21c87783f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"35820c96-5f2a-4e2e-bc4b-714ab8e1c841","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"da975ea1-eeb6-4619-b73f-786b46dd8fe6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T01:55:09.220 DEBUG:teuthology.misc:0 of 4 OSDs are up 2024-04-24T01:55:09.264 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-24T01:55:09.263+0000 7f0022d6d000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-24T01:55:09.529 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-24T01:55:09.529+0000 7f0022d6d000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-24T01:55:09.738 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-24T01:55:09.737+0000 7f0022d6d000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-24T01:55:09.821 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-24T01:55:09.820+0000 7f0022d6d000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-24T01:55:09.937 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-24T01:55:09.937+0000 7f0022d6d000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-24T01:55:10.009 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-24T01:55:10.008+0000 7f0022d6d000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-24T01:55:10.783 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-24T01:55:10.782+0000 7f0022d6d000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-24T01:55:10.927 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-24T01:55:10.926+0000 7f0022d6d000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-24T01:55:11.936 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-24T01:55:11.935+0000 7effc1e7b640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-04-24T01:55:15.624 DEBUG:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-24T01:55:15.735 INFO:teuthology.misc.health.smithi079.stderr:2024-04-24T01:55:15.730+0000 7f5265aed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:15.739 INFO:teuthology.misc.health.smithi079.stderr:2024-04-24T01:55:15.738+0000 7f5265aed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:15.927 INFO:teuthology.misc.health.smithi079.stdout: 2024-04-24T01:55:15.928 INFO:teuthology.misc.health.smithi079.stdout:{"epoch":11,"fsid":"e600dd1c-0bef-4ea0-91c9-2f649f75622f","created":"2024-04-24T01:55:06.017534+0000","modified":"2024-04-24T01:55:15.127115+0000","last_up_change":"2024-04-24T01:55:14.120010+0000","last_in_change":"2024-04-24T01:55:08.126530+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T01:55:14.952331+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.75,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1b20709f-f1e7-4d59-8402-0d13620f747e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6800","nonce":3830851766}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6803","nonce":3830851766}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6805","nonce":3830851766}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":3830851766}]},"public_addr":"172.21.15.79:6800/3830851766","cluster_addr":"172.21.15.79:6803/3830851766","heartbeat_back_addr":"172.21.15.79:6805/3830851766","heartbeat_front_addr":"172.21.15.79:6808/3830851766","state":["exists","up"]},{"osd":1,"uuid":"54fd0eee-6d68-4c92-9ac7-6b21c87783f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6801","nonce":1823728464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":1823728464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":1823728464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6807","nonce":1823728464}]},"public_addr":"172.21.15.79:6801/1823728464","cluster_addr":"172.21.15.79:6802/1823728464","heartbeat_back_addr":"172.21.15.79:6806/1823728464","heartbeat_front_addr":"172.21.15.79:6807/1823728464","state":["exists","up"]},{"osd":2,"uuid":"35820c96-5f2a-4e2e-bc4b-714ab8e1c841","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6803","nonce":2585849279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":2585849279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":2585849279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6817","nonce":2585849279}]},"public_addr":"172.21.15.103:6803/2585849279","cluster_addr":"172.21.15.103:6804/2585849279","heartbeat_back_addr":"172.21.15.103:6816/2585849279","heartbeat_front_addr":"172.21.15.103:6817/2585849279","state":["exists","up"]},{"osd":3,"uuid":"da975ea1-eeb6-4619-b73f-786b46dd8fe6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6809","nonce":659350107}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":659350107}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":659350107}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6811","nonce":659350107}]},"public_addr":"172.21.15.103:6809/659350107","cluster_addr":"172.21.15.103:6808/659350107","heartbeat_back_addr":"172.21.15.103:6812/659350107","heartbeat_front_addr":"172.21.15.103:6811/659350107","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T01:55:15.939 DEBUG:teuthology.misc:4 of 4 OSDs are up 2024-04-24T01:55:15.939 INFO:tasks.ceph:Creating RBD pool 2024-04-24T01:55:15.939 DEBUG:teuthology.orchestra.run.smithi079:> sudo ceph --cluster ceph osd pool create rbd 8 2024-04-24T01:55:16.045 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:16.037+0000 7fd90bc24640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:16.049 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:16.048+0000 7fd90bc24640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:17.137 INFO:teuthology.orchestra.run.smithi079.stderr:pool 'rbd' created 2024-04-24T01:55:17.155 DEBUG:teuthology.orchestra.run.smithi079:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2024-04-24T01:55:17.299 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:17.288+0000 7f861106d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:17.302 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:17.302+0000 7f861106d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:18.138 INFO:teuthology.orchestra.run.smithi079.stderr:enabled application 'rbd' on pool 'rbd' 2024-04-24T01:55:18.157 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-04-24T01:55:18.157 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-24T01:55:18.158 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-04-24T01:55:18.158 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-04-24T01:55:18.158 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-04-24T01:55:18.305 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:18.300+0000 7f0d684ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:18.309 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:18.308+0000 7f0d684ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:18.547 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:55:18.558 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":4,"flags":0,"active_gid":4099,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":1617922284},{"type":"v1","addr":"172.21.15.103:6801","nonce":1617922284}]},"active_addr":"172.21.15.103:6801/1617922284","active_change":"2024-04-24T01:55:10.929671+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.103:0","nonce":1207611050}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.103:0","nonce":541022587}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.103:0","nonce":1269307670}]}]} 2024-04-24T01:55:18.562 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-04-24T01:55:18.562 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-04-24T01:55:18.563 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T01:55:18.669 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:18.665+0000 7f6b7099d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:18.673 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:18.672+0000 7f6b7099d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:18.864 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:55:18.864 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":14,"fsid":"e600dd1c-0bef-4ea0-91c9-2f649f75622f","created":"2024-04-24T01:55:06.017534+0000","modified":"2024-04-24T01:55:18.137508+0000","last_up_change":"2024-04-24T01:55:14.120010+0000","last_in_change":"2024-04-24T01:55:08.126530+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T01:55:14.952331+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.75,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-24T01:55:16.243536+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1b20709f-f1e7-4d59-8402-0d13620f747e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6800","nonce":3830851766}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6803","nonce":3830851766}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6805","nonce":3830851766}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":3830851766}]},"public_addr":"172.21.15.79:6800/3830851766","cluster_addr":"172.21.15.79:6803/3830851766","heartbeat_back_addr":"172.21.15.79:6805/3830851766","heartbeat_front_addr":"172.21.15.79:6808/3830851766","state":["exists","up"]},{"osd":1,"uuid":"54fd0eee-6d68-4c92-9ac7-6b21c87783f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6801","nonce":1823728464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":1823728464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":1823728464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6807","nonce":1823728464}]},"public_addr":"172.21.15.79:6801/1823728464","cluster_addr":"172.21.15.79:6802/1823728464","heartbeat_back_addr":"172.21.15.79:6806/1823728464","heartbeat_front_addr":"172.21.15.79:6807/1823728464","state":["exists","up"]},{"osd":2,"uuid":"35820c96-5f2a-4e2e-bc4b-714ab8e1c841","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6803","nonce":2585849279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":2585849279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":2585849279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6817","nonce":2585849279}]},"public_addr":"172.21.15.103:6803/2585849279","cluster_addr":"172.21.15.103:6804/2585849279","heartbeat_back_addr":"172.21.15.103:6816/2585849279","heartbeat_front_addr":"172.21.15.103:6817/2585849279","state":["exists","up"]},{"osd":3,"uuid":"da975ea1-eeb6-4619-b73f-786b46dd8fe6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6809","nonce":659350107}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":659350107}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":659350107}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6811","nonce":659350107}]},"public_addr":"172.21.15.103:6809/659350107","cluster_addr":"172.21.15.103:6808/659350107","heartbeat_back_addr":"172.21.15.103:6812/659350107","heartbeat_front_addr":"172.21.15.103:6811/659350107","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T01:55:18.876 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-04-24T01:55:18.876 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T01:55:18.983 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:18.979+0000 7fac5acad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:18.987 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:18.986+0000 7fac5acad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:19.179 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:55:19.179 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":15,"fsid":"e600dd1c-0bef-4ea0-91c9-2f649f75622f","created":"2024-04-24T01:55:06.017534+0000","modified":"2024-04-24T01:55:19.141942+0000","last_up_change":"2024-04-24T01:55:14.120010+0000","last_in_change":"2024-04-24T01:55:08.126530+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T01:55:14.952331+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.75,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-24T01:55:16.243536+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1b20709f-f1e7-4d59-8402-0d13620f747e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6800","nonce":3830851766}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6803","nonce":3830851766}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6805","nonce":3830851766}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":3830851766}]},"public_addr":"172.21.15.79:6800/3830851766","cluster_addr":"172.21.15.79:6803/3830851766","heartbeat_back_addr":"172.21.15.79:6805/3830851766","heartbeat_front_addr":"172.21.15.79:6808/3830851766","state":["exists","up"]},{"osd":1,"uuid":"54fd0eee-6d68-4c92-9ac7-6b21c87783f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6801","nonce":1823728464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":1823728464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":1823728464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6807","nonce":1823728464}]},"public_addr":"172.21.15.79:6801/1823728464","cluster_addr":"172.21.15.79:6802/1823728464","heartbeat_back_addr":"172.21.15.79:6806/1823728464","heartbeat_front_addr":"172.21.15.79:6807/1823728464","state":["exists","up"]},{"osd":2,"uuid":"35820c96-5f2a-4e2e-bc4b-714ab8e1c841","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6803","nonce":2585849279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":2585849279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":2585849279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6817","nonce":2585849279}]},"public_addr":"172.21.15.103:6803/2585849279","cluster_addr":"172.21.15.103:6804/2585849279","heartbeat_back_addr":"172.21.15.103:6816/2585849279","heartbeat_front_addr":"172.21.15.103:6817/2585849279","state":["exists","up"]},{"osd":3,"uuid":"da975ea1-eeb6-4619-b73f-786b46dd8fe6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6809","nonce":659350107}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":659350107}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":659350107}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6811","nonce":659350107}]},"public_addr":"172.21.15.103:6809/659350107","cluster_addr":"172.21.15.103:6808/659350107","heartbeat_back_addr":"172.21.15.103:6812/659350107","heartbeat_front_addr":"172.21.15.103:6811/659350107","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T01:55:19.192 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-04-24T01:55:19.193 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-04-24T01:55:19.193 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-04-24T01:55:19.194 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-04-24T01:55:19.310 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:19.304+0000 7fb59726d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:19.311 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:19.304+0000 7fee3806d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:19.311 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:19.305+0000 7f9ae4aad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:19.311 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:19.304+0000 7fe95202d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:19.315 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:19.314+0000 7fee3806d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:19.315 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:19.314+0000 7fb59726d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:19.316 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:19.315+0000 7f9ae4aad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:19.316 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:19.315+0000 7fe95202d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:19.347 INFO:teuthology.orchestra.run.smithi079.stdout:42949672965 2024-04-24T01:55:19.347 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-24T01:55:19.351 INFO:teuthology.orchestra.run.smithi079.stdout:38654705670 2024-04-24T01:55:19.351 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-24T01:55:19.353 INFO:teuthology.orchestra.run.smithi079.stdout:42949672965 2024-04-24T01:55:19.354 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-24T01:55:19.355 INFO:teuthology.orchestra.run.smithi079.stdout:38654705670 2024-04-24T01:55:19.355 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-24T01:55:19.458 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:19.452+0000 7f14f4ead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:19.462 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:19.462+0000 7f14f4ead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:19.508 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:19.502+0000 7fa83cf9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:19.510 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:19.504+0000 7f2021c64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:19.513 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:19.512+0000 7fa83cf9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:19.514 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:19.514+0000 7f2021c64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:19.523 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:19.514+0000 7f8c666ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:19.533 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:19.532+0000 7f8c666ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:19.718 INFO:teuthology.orchestra.run.smithi079.stdout:0 2024-04-24T01:55:19.729 INFO:teuthology.orchestra.run.smithi079.stdout:0 2024-04-24T01:55:19.731 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.0 2024-04-24T01:55:19.731 INFO:teuthology.orchestra.run.smithi079.stdout:0 2024-04-24T01:55:19.744 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672965 got 0 for osd.2 2024-04-24T01:55:19.746 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672965 got 0 for osd.3 2024-04-24T01:55:19.764 INFO:teuthology.orchestra.run.smithi079.stdout:0 2024-04-24T01:55:19.777 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2024-04-24T01:55:20.731 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-24T01:55:20.745 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-24T01:55:20.747 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-24T01:55:20.777 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-24T01:55:20.859 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:20.851+0000 7fa7a9aed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:20.864 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:20.863+0000 7fa7a9aed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:20.895 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:20.888+0000 7fc5e1ead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:20.897 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:20.890+0000 7f0700b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:20.899 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:20.899+0000 7fc5e1ead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:20.902 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:20.901+0000 7f0700b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:20.945 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:20.939+0000 7f2f50b1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:20.950 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:20.949+0000 7f2f50b1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:21.075 INFO:teuthology.orchestra.run.smithi079.stdout:0 2024-04-24T01:55:21.089 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.0 2024-04-24T01:55:21.109 INFO:teuthology.orchestra.run.smithi079.stdout:0 2024-04-24T01:55:21.116 INFO:teuthology.orchestra.run.smithi079.stdout:0 2024-04-24T01:55:21.122 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672965 got 0 for osd.3 2024-04-24T01:55:21.130 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672965 got 0 for osd.2 2024-04-24T01:55:21.153 INFO:teuthology.orchestra.run.smithi079.stdout:0 2024-04-24T01:55:21.166 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2024-04-24T01:55:22.090 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-24T01:55:22.123 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-24T01:55:22.131 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-24T01:55:22.167 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-24T01:55:22.206 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:22.200+0000 7fe5f8f5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:22.212 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:22.211+0000 7fe5f8f5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:22.253 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:22.247+0000 7f79c5b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:22.257 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:22.257+0000 7f79c5b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:22.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:22.259+0000 7f43b8a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:22.275 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:22.274+0000 7f43b8a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:22.308 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:22.302+0000 7f433fe6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:22.312 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:22.311+0000 7f433fe6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:22.473 INFO:teuthology.orchestra.run.smithi079.stdout:0 2024-04-24T01:55:22.487 INFO:teuthology.orchestra.run.smithi079.stdout:0 2024-04-24T01:55:22.491 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672965 got 0 for osd.3 2024-04-24T01:55:22.500 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.0 2024-04-24T01:55:22.509 INFO:teuthology.orchestra.run.smithi079.stdout:0 2024-04-24T01:55:22.511 INFO:teuthology.orchestra.run.smithi079.stdout:0 2024-04-24T01:55:22.523 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2024-04-24T01:55:22.524 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672965 got 0 for osd.2 2024-04-24T01:55:23.492 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-24T01:55:23.501 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-24T01:55:23.524 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-24T01:55:23.525 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-24T01:55:23.611 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:23.603+0000 7fee41e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:23.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:23.617+0000 7fee41e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:23.652 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:23.647+0000 7f5a0c4ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:23.657 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:23.651+0000 7f2d6379d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:23.657 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:23.656+0000 7f5a0c4ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:23.657 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:23.651+0000 7f8694a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:23.662 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:23.661+0000 7f2d6379d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:23.662 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:23.661+0000 7f8694a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:23.827 INFO:teuthology.orchestra.run.smithi079.stdout:0 2024-04-24T01:55:23.845 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672965 got 0 for osd.3 2024-04-24T01:55:23.873 INFO:teuthology.orchestra.run.smithi079.stdout:0 2024-04-24T01:55:23.879 INFO:teuthology.orchestra.run.smithi079.stdout:0 2024-04-24T01:55:23.887 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.0 2024-04-24T01:55:23.887 INFO:teuthology.orchestra.run.smithi079.stdout:0 2024-04-24T01:55:23.893 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672965 got 0 for osd.2 2024-04-24T01:55:23.900 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2024-04-24T01:55:24.846 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-24T01:55:24.888 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-24T01:55:24.894 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-24T01:55:24.902 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-24T01:55:24.960 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:24.953+0000 7fbf7e95d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:24.965 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:24.964+0000 7fbf7e95d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:25.003 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:24.997+0000 7fea7dead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:25.008 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:25.007+0000 7fea7dead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:25.049 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:25.043+0000 7f13fc59d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:25.053 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:25.052+0000 7f13fc59d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:25.072 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:25.067+0000 7f6363c2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:25.077 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:25.076+0000 7f6363c2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:25.213 INFO:teuthology.orchestra.run.smithi079.stdout:38654705675 2024-04-24T01:55:25.217 INFO:teuthology.orchestra.run.smithi079.stdout:42949672970 2024-04-24T01:55:25.226 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705675 for osd.0 2024-04-24T01:55:25.227 DEBUG:teuthology.parallel:result is None 2024-04-24T01:55:25.235 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672965 got 42949672970 for osd.3 2024-04-24T01:55:25.235 DEBUG:teuthology.parallel:result is None 2024-04-24T01:55:25.267 INFO:teuthology.orchestra.run.smithi079.stdout:42949672970 2024-04-24T01:55:25.280 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672965 got 42949672970 for osd.2 2024-04-24T01:55:25.280 DEBUG:teuthology.parallel:result is None 2024-04-24T01:55:25.286 INFO:teuthology.orchestra.run.smithi079.stdout:38654705674 2024-04-24T01:55:25.299 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705674 for osd.1 2024-04-24T01:55:25.299 DEBUG:teuthology.parallel:result is None 2024-04-24T01:55:25.299 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-24T01:55:25.299 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T01:55:25.446 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:25.442+0000 7f32bf2ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:25.450 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:25.449+0000 7f32bf2ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:25.634 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:55:25.634 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-24T01:55:25.646 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2024-04-24T01:55:24.935064+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":107980,"kb_used_data":660,"kb_used_omap":0,"kb_used_meta":107264,"kb_avail":377379380,"statfs":{"total":386547056640,"available":386436485120,"internally_reserved":0,"allocated":675840,"data_stored":1895372,"data_compressed":89889,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109838336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.778942"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-24T01:55:18.239215+0000","last_change":"2024-04-24T01:55:18.241117+0000","last_active":"2024-04-24T01:55:18.239215+0000","last_peered":"2024-04-24T01:55:18.239215+0000","last_clean":"2024-04-24T01:55:17.134477+0000","last_became_active":"2024-04-24T01:55:18.234313+0000","last_became_peered":"2024-04-24T01:55:18.234313+0000","last_unstale":"2024-04-24T01:55:18.239215+0000","last_undegraded":"2024-04-24T01:55:18.239215+0000","last_fullsized":"2024-04-24T01:55:18.239215+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-24T01:55:19.165132+0000","last_change":"2024-04-24T01:55:18.278351+0000","last_active":"2024-04-24T01:55:19.165132+0000","last_peered":"2024-04-24T01:55:19.165132+0000","last_clean":"2024-04-24T01:55:19.165132+0000","last_became_active":"2024-04-24T01:55:18.273668+0000","last_became_peered":"2024-04-24T01:55:18.273668+0000","last_unstale":"2024-04-24T01:55:19.165132+0000","last_undegraded":"2024-04-24T01:55:19.165132+0000","last_fullsized":"2024-04-24T01:55:19.165132+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-24T01:55:18.234008+0000","last_change":"2024-04-24T01:55:18.235763+0000","last_active":"2024-04-24T01:55:18.234008+0000","last_peered":"2024-04-24T01:55:18.234008+0000","last_clean":"2024-04-24T01:55:17.134477+0000","last_became_active":"2024-04-24T01:55:18.229175+0000","last_became_peered":"2024-04-24T01:55:18.229175+0000","last_unstale":"2024-04-24T01:55:18.234008+0000","last_undegraded":"2024-04-24T01:55:18.234008+0000","last_fullsized":"2024-04-24T01:55:18.234008+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-24T01:55:19.164444+0000","last_change":"2024-04-24T01:55:18.224238+0000","last_active":"2024-04-24T01:55:19.164444+0000","last_peered":"2024-04-24T01:55:19.164444+0000","last_clean":"2024-04-24T01:55:19.164444+0000","last_became_active":"2024-04-24T01:55:18.219131+0000","last_became_peered":"2024-04-24T01:55:18.219131+0000","last_unstale":"2024-04-24T01:55:19.164444+0000","last_undegraded":"2024-04-24T01:55:19.164444+0000","last_fullsized":"2024-04-24T01:55:19.164444+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-24T01:55:24.146605+0000","last_change":"2024-04-24T01:55:18.230893+0000","last_active":"2024-04-24T01:55:24.146605+0000","last_peered":"2024-04-24T01:55:24.146605+0000","last_clean":"2024-04-24T01:55:24.146605+0000","last_became_active":"2024-04-24T01:55:18.225188+0000","last_became_peered":"2024-04-24T01:55:18.225188+0000","last_unstale":"2024-04-24T01:55:24.146605+0000","last_undegraded":"2024-04-24T01:55:24.146605+0000","last_fullsized":"2024-04-24T01:55:24.146605+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-24T01:55:18.213174+0000","last_change":"2024-04-24T01:55:18.215251+0000","last_active":"2024-04-24T01:55:18.213174+0000","last_peered":"2024-04-24T01:55:18.213174+0000","last_clean":"2024-04-24T01:55:17.134477+0000","last_became_active":"2024-04-24T01:55:18.206605+0000","last_became_peered":"2024-04-24T01:55:18.206605+0000","last_unstale":"2024-04-24T01:55:18.213174+0000","last_undegraded":"2024-04-24T01:55:18.213174+0000","last_fullsized":"2024-04-24T01:55:18.213174+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-24T01:55:24.148481+0000","last_change":"2024-04-24T01:55:18.265905+0000","last_active":"2024-04-24T01:55:24.148481+0000","last_peered":"2024-04-24T01:55:24.148481+0000","last_clean":"2024-04-24T01:55:24.148481+0000","last_became_active":"2024-04-24T01:55:18.257020+0000","last_became_peered":"2024-04-24T01:55:18.257020+0000","last_unstale":"2024-04-24T01:55:24.148481+0000","last_undegraded":"2024-04-24T01:55:24.148481+0000","last_fullsized":"2024-04-24T01:55:24.148481+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-24T01:55:18.234590+0000","last_change":"2024-04-24T01:55:18.236148+0000","last_active":"2024-04-24T01:55:18.234590+0000","last_peered":"2024-04-24T01:55:18.234590+0000","last_clean":"2024-04-24T01:55:17.134477+0000","last_became_active":"2024-04-24T01:55:18.230154+0000","last_became_peered":"2024-04-24T01:55:18.230154+0000","last_unstale":"2024-04-24T01:55:18.234590+0000","last_undegraded":"2024-04-24T01:55:18.234590+0000","last_fullsized":"2024-04-24T01:55:18.234590+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'97","reported_seq":142,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-24T01:55:18.148759+0000","last_change":"2024-04-24T01:55:16.196470+0000","last_active":"2024-04-24T01:55:18.148759+0000","last_peered":"2024-04-24T01:55:18.148759+0000","last_clean":"2024-04-24T01:55:18.148759+0000","last_became_active":"2024-04-24T01:55:16.190464+0000","last_became_peered":"2024-04-24T01:55:16.190464+0000","last_unstale":"2024-04-24T01:55:18.148759+0000","last_undegraded":"2024-04-24T01:55:18.148759+0000","last_fullsized":"2024-04-24T01:55:18.148759+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:15.127115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:15.127115+0000","last_clean_scrub_stamp":"2024-04-24T01:55:15.127115+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":1}],"osd_stats":[{"osd":3,"up_from":10,"seq":42949672970,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27012,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344828,"statfs":{"total":96636764160,"available":96609103872,"internally_reserved":0,"allocated":180224,"data_stored":619830,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949672970,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27020,"kb_used_data":184,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344820,"statfs":{"total":96636764160,"available":96609095680,"internally_reserved":0,"allocated":188416,"data_stored":623040,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705675,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27008,"kb_used_data":184,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344832,"statfs":{"total":96636764160,"available":96609107968,"internally_reserved":0,"allocated":188416,"data_stored":623040,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26940,"kb_used_data":116,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344900,"statfs":{"total":96636764160,"available":96609177600,"internally_reserved":0,"allocated":118784,"data_stored":29462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T01:55:25.647 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T01:55:25.754 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:25.748+0000 7f8b7539d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:25.757 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:25.756+0000 7f8b7539d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:25.942 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:55:25.943 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-24T01:55:25.955 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2024-04-24T01:55:24.935064+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":107980,"kb_used_data":660,"kb_used_omap":0,"kb_used_meta":107264,"kb_avail":377379380,"statfs":{"total":386547056640,"available":386436485120,"internally_reserved":0,"allocated":675840,"data_stored":1895372,"data_compressed":89889,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109838336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.778942"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-24T01:55:18.239215+0000","last_change":"2024-04-24T01:55:18.241117+0000","last_active":"2024-04-24T01:55:18.239215+0000","last_peered":"2024-04-24T01:55:18.239215+0000","last_clean":"2024-04-24T01:55:17.134477+0000","last_became_active":"2024-04-24T01:55:18.234313+0000","last_became_peered":"2024-04-24T01:55:18.234313+0000","last_unstale":"2024-04-24T01:55:18.239215+0000","last_undegraded":"2024-04-24T01:55:18.239215+0000","last_fullsized":"2024-04-24T01:55:18.239215+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-24T01:55:19.165132+0000","last_change":"2024-04-24T01:55:18.278351+0000","last_active":"2024-04-24T01:55:19.165132+0000","last_peered":"2024-04-24T01:55:19.165132+0000","last_clean":"2024-04-24T01:55:19.165132+0000","last_became_active":"2024-04-24T01:55:18.273668+0000","last_became_peered":"2024-04-24T01:55:18.273668+0000","last_unstale":"2024-04-24T01:55:19.165132+0000","last_undegraded":"2024-04-24T01:55:19.165132+0000","last_fullsized":"2024-04-24T01:55:19.165132+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-24T01:55:18.234008+0000","last_change":"2024-04-24T01:55:18.235763+0000","last_active":"2024-04-24T01:55:18.234008+0000","last_peered":"2024-04-24T01:55:18.234008+0000","last_clean":"2024-04-24T01:55:17.134477+0000","last_became_active":"2024-04-24T01:55:18.229175+0000","last_became_peered":"2024-04-24T01:55:18.229175+0000","last_unstale":"2024-04-24T01:55:18.234008+0000","last_undegraded":"2024-04-24T01:55:18.234008+0000","last_fullsized":"2024-04-24T01:55:18.234008+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-24T01:55:19.164444+0000","last_change":"2024-04-24T01:55:18.224238+0000","last_active":"2024-04-24T01:55:19.164444+0000","last_peered":"2024-04-24T01:55:19.164444+0000","last_clean":"2024-04-24T01:55:19.164444+0000","last_became_active":"2024-04-24T01:55:18.219131+0000","last_became_peered":"2024-04-24T01:55:18.219131+0000","last_unstale":"2024-04-24T01:55:19.164444+0000","last_undegraded":"2024-04-24T01:55:19.164444+0000","last_fullsized":"2024-04-24T01:55:19.164444+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-24T01:55:24.146605+0000","last_change":"2024-04-24T01:55:18.230893+0000","last_active":"2024-04-24T01:55:24.146605+0000","last_peered":"2024-04-24T01:55:24.146605+0000","last_clean":"2024-04-24T01:55:24.146605+0000","last_became_active":"2024-04-24T01:55:18.225188+0000","last_became_peered":"2024-04-24T01:55:18.225188+0000","last_unstale":"2024-04-24T01:55:24.146605+0000","last_undegraded":"2024-04-24T01:55:24.146605+0000","last_fullsized":"2024-04-24T01:55:24.146605+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-24T01:55:18.213174+0000","last_change":"2024-04-24T01:55:18.215251+0000","last_active":"2024-04-24T01:55:18.213174+0000","last_peered":"2024-04-24T01:55:18.213174+0000","last_clean":"2024-04-24T01:55:17.134477+0000","last_became_active":"2024-04-24T01:55:18.206605+0000","last_became_peered":"2024-04-24T01:55:18.206605+0000","last_unstale":"2024-04-24T01:55:18.213174+0000","last_undegraded":"2024-04-24T01:55:18.213174+0000","last_fullsized":"2024-04-24T01:55:18.213174+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-24T01:55:24.148481+0000","last_change":"2024-04-24T01:55:18.265905+0000","last_active":"2024-04-24T01:55:24.148481+0000","last_peered":"2024-04-24T01:55:24.148481+0000","last_clean":"2024-04-24T01:55:24.148481+0000","last_became_active":"2024-04-24T01:55:18.257020+0000","last_became_peered":"2024-04-24T01:55:18.257020+0000","last_unstale":"2024-04-24T01:55:24.148481+0000","last_undegraded":"2024-04-24T01:55:24.148481+0000","last_fullsized":"2024-04-24T01:55:24.148481+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-24T01:55:18.234590+0000","last_change":"2024-04-24T01:55:18.236148+0000","last_active":"2024-04-24T01:55:18.234590+0000","last_peered":"2024-04-24T01:55:18.234590+0000","last_clean":"2024-04-24T01:55:17.134477+0000","last_became_active":"2024-04-24T01:55:18.230154+0000","last_became_peered":"2024-04-24T01:55:18.230154+0000","last_unstale":"2024-04-24T01:55:18.234590+0000","last_undegraded":"2024-04-24T01:55:18.234590+0000","last_fullsized":"2024-04-24T01:55:18.234590+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'97","reported_seq":142,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-24T01:55:18.148759+0000","last_change":"2024-04-24T01:55:16.196470+0000","last_active":"2024-04-24T01:55:18.148759+0000","last_peered":"2024-04-24T01:55:18.148759+0000","last_clean":"2024-04-24T01:55:18.148759+0000","last_became_active":"2024-04-24T01:55:16.190464+0000","last_became_peered":"2024-04-24T01:55:16.190464+0000","last_unstale":"2024-04-24T01:55:18.148759+0000","last_undegraded":"2024-04-24T01:55:18.148759+0000","last_fullsized":"2024-04-24T01:55:18.148759+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:15.127115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:15.127115+0000","last_clean_scrub_stamp":"2024-04-24T01:55:15.127115+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":1}],"osd_stats":[{"osd":3,"up_from":10,"seq":42949672970,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27012,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344828,"statfs":{"total":96636764160,"available":96609103872,"internally_reserved":0,"allocated":180224,"data_stored":619830,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949672970,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27020,"kb_used_data":184,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344820,"statfs":{"total":96636764160,"available":96609095680,"internally_reserved":0,"allocated":188416,"data_stored":623040,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705675,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27008,"kb_used_data":184,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344832,"statfs":{"total":96636764160,"available":96609107968,"internally_reserved":0,"allocated":188416,"data_stored":623040,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26940,"kb_used_data":116,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344900,"statfs":{"total":96636764160,"available":96609177600,"internally_reserved":0,"allocated":118784,"data_stored":29462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T01:55:25.956 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-24T01:55:25.956 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-24T01:55:25.956 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-04-24T01:55:25.956 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-04-24T01:55:26.062 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:26.058+0000 7f741611d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:26.066 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:26.065+0000 7f741611d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:26.304 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:55:26.305 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-24T01:55:26.317 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-04-24T01:55:26.318 INFO:teuthology.run_tasks:Running task thrashosds... 2024-04-24T01:55:26.329 INFO:tasks.thrashosds:config is {'ceph_objectstore_tool': False, 'chance_down': 0, 'chance_inject_pause_short': 0, 'chance_reset_purged_snaps_last': 0, 'chance_thrash_cluster_full': 0, 'chance_thrash_pg_upmap': 0, 'dump_ops_enable': False, 'min_in': 3, 'noscrub_toggle_delay': 0, 'reweight_osd': 0, 'sighup_delay': 0, 'thrash_primary_affinity': False, 'timeout': 2400, 'optrack_toggle_delay': 2.0, 'random_eio': 0.0} 2024-04-24T01:55:26.329 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'ci-yingxin-crimson-load-obc-new-1'}, 'ceph': {'conf': {'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum'], 'sha1': '424d8c4389b684b0a4d72510e1b4d398d8e6c8f4'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'flavor': 'crimson', 'sha1': '424d8c4389b684b0a4d72510e1b4d398d8e6c8f4'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'thrashosds': {'min_in': 4}, 'workunit': {'branch': 'ci-yingxin-crimson-load-obc-new-1', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': '424d8c4389b684b0a4d72510e1b4d398d8e6c8f4'}} 2024-04-24T01:55:26.329 INFO:tasks.thrashosds:config is {'ceph_objectstore_tool': False, 'chance_down': 0, 'chance_inject_pause_short': 0, 'chance_reset_purged_snaps_last': 0, 'chance_thrash_cluster_full': 0, 'chance_thrash_pg_upmap': 0, 'dump_ops_enable': False, 'min_in': 4, 'noscrub_toggle_delay': 0, 'reweight_osd': 0, 'sighup_delay': 0, 'thrash_primary_affinity': False, 'timeout': 2400, 'optrack_toggle_delay': 2.0, 'random_eio': 0.0} 2024-04-24T01:55:26.329 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-04-24T01:55:26.440 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:26.432+0000 7f7890c2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:26.443 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:26.442+0000 7f7890c2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:26.644 INFO:tasks.thrashosds:Beginning thrashosds... 2024-04-24T01:55:26.644 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-24T01:55:26.644 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T01:55:26.750 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:26.746+0000 7fb3eaaad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:26.754 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:26.753+0000 7fb3eaaad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:26.939 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:55:26.939 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-24T01:55:26.951 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2024-04-24T01:55:26.935506+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":107980,"kb_used_data":660,"kb_used_omap":0,"kb_used_meta":107264,"kb_avail":377379380,"statfs":{"total":386547056640,"available":386436485120,"internally_reserved":0,"allocated":675840,"data_stored":1895372,"data_compressed":89889,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109838336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.779384"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-24T01:55:18.239215+0000","last_change":"2024-04-24T01:55:18.241117+0000","last_active":"2024-04-24T01:55:18.239215+0000","last_peered":"2024-04-24T01:55:18.239215+0000","last_clean":"2024-04-24T01:55:17.134477+0000","last_became_active":"2024-04-24T01:55:18.234313+0000","last_became_peered":"2024-04-24T01:55:18.234313+0000","last_unstale":"2024-04-24T01:55:18.239215+0000","last_undegraded":"2024-04-24T01:55:18.239215+0000","last_fullsized":"2024-04-24T01:55:18.239215+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-24T01:55:19.165132+0000","last_change":"2024-04-24T01:55:18.278351+0000","last_active":"2024-04-24T01:55:19.165132+0000","last_peered":"2024-04-24T01:55:19.165132+0000","last_clean":"2024-04-24T01:55:19.165132+0000","last_became_active":"2024-04-24T01:55:18.273668+0000","last_became_peered":"2024-04-24T01:55:18.273668+0000","last_unstale":"2024-04-24T01:55:19.165132+0000","last_undegraded":"2024-04-24T01:55:19.165132+0000","last_fullsized":"2024-04-24T01:55:19.165132+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-24T01:55:18.234008+0000","last_change":"2024-04-24T01:55:18.235763+0000","last_active":"2024-04-24T01:55:18.234008+0000","last_peered":"2024-04-24T01:55:18.234008+0000","last_clean":"2024-04-24T01:55:17.134477+0000","last_became_active":"2024-04-24T01:55:18.229175+0000","last_became_peered":"2024-04-24T01:55:18.229175+0000","last_unstale":"2024-04-24T01:55:18.234008+0000","last_undegraded":"2024-04-24T01:55:18.234008+0000","last_fullsized":"2024-04-24T01:55:18.234008+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-24T01:55:19.164444+0000","last_change":"2024-04-24T01:55:18.224238+0000","last_active":"2024-04-24T01:55:19.164444+0000","last_peered":"2024-04-24T01:55:19.164444+0000","last_clean":"2024-04-24T01:55:19.164444+0000","last_became_active":"2024-04-24T01:55:18.219131+0000","last_became_peered":"2024-04-24T01:55:18.219131+0000","last_unstale":"2024-04-24T01:55:19.164444+0000","last_undegraded":"2024-04-24T01:55:19.164444+0000","last_fullsized":"2024-04-24T01:55:19.164444+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-24T01:55:24.146605+0000","last_change":"2024-04-24T01:55:18.230893+0000","last_active":"2024-04-24T01:55:24.146605+0000","last_peered":"2024-04-24T01:55:24.146605+0000","last_clean":"2024-04-24T01:55:24.146605+0000","last_became_active":"2024-04-24T01:55:18.225188+0000","last_became_peered":"2024-04-24T01:55:18.225188+0000","last_unstale":"2024-04-24T01:55:24.146605+0000","last_undegraded":"2024-04-24T01:55:24.146605+0000","last_fullsized":"2024-04-24T01:55:24.146605+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-24T01:55:18.213174+0000","last_change":"2024-04-24T01:55:18.215251+0000","last_active":"2024-04-24T01:55:18.213174+0000","last_peered":"2024-04-24T01:55:18.213174+0000","last_clean":"2024-04-24T01:55:17.134477+0000","last_became_active":"2024-04-24T01:55:18.206605+0000","last_became_peered":"2024-04-24T01:55:18.206605+0000","last_unstale":"2024-04-24T01:55:18.213174+0000","last_undegraded":"2024-04-24T01:55:18.213174+0000","last_fullsized":"2024-04-24T01:55:18.213174+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-24T01:55:24.148481+0000","last_change":"2024-04-24T01:55:18.265905+0000","last_active":"2024-04-24T01:55:24.148481+0000","last_peered":"2024-04-24T01:55:24.148481+0000","last_clean":"2024-04-24T01:55:24.148481+0000","last_became_active":"2024-04-24T01:55:18.257020+0000","last_became_peered":"2024-04-24T01:55:18.257020+0000","last_unstale":"2024-04-24T01:55:24.148481+0000","last_undegraded":"2024-04-24T01:55:24.148481+0000","last_fullsized":"2024-04-24T01:55:24.148481+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-24T01:55:18.234590+0000","last_change":"2024-04-24T01:55:18.236148+0000","last_active":"2024-04-24T01:55:18.234590+0000","last_peered":"2024-04-24T01:55:18.234590+0000","last_clean":"2024-04-24T01:55:17.134477+0000","last_became_active":"2024-04-24T01:55:18.230154+0000","last_became_peered":"2024-04-24T01:55:18.230154+0000","last_unstale":"2024-04-24T01:55:18.234590+0000","last_undegraded":"2024-04-24T01:55:18.234590+0000","last_fullsized":"2024-04-24T01:55:18.234590+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'97","reported_seq":142,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-24T01:55:18.148759+0000","last_change":"2024-04-24T01:55:16.196470+0000","last_active":"2024-04-24T01:55:18.148759+0000","last_peered":"2024-04-24T01:55:18.148759+0000","last_clean":"2024-04-24T01:55:18.148759+0000","last_became_active":"2024-04-24T01:55:16.190464+0000","last_became_peered":"2024-04-24T01:55:16.190464+0000","last_unstale":"2024-04-24T01:55:18.148759+0000","last_undegraded":"2024-04-24T01:55:18.148759+0000","last_fullsized":"2024-04-24T01:55:18.148759+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:15.127115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:15.127115+0000","last_clean_scrub_stamp":"2024-04-24T01:55:15.127115+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":1}],"osd_stats":[{"osd":3,"up_from":10,"seq":42949672970,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27012,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344828,"statfs":{"total":96636764160,"available":96609103872,"internally_reserved":0,"allocated":180224,"data_stored":619830,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949672970,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27020,"kb_used_data":184,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344820,"statfs":{"total":96636764160,"available":96609095680,"internally_reserved":0,"allocated":188416,"data_stored":623040,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705675,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27008,"kb_used_data":184,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344832,"statfs":{"total":96636764160,"available":96609107968,"internally_reserved":0,"allocated":188416,"data_stored":623040,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26940,"kb_used_data":116,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344900,"statfs":{"total":96636764160,"available":96609177600,"internally_reserved":0,"allocated":118784,"data_stored":29462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T01:55:26.952 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T01:55:27.058 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:27.054+0000 7fb51539d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:27.062 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:27.061+0000 7fb51539d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:27.248 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:55:27.248 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-24T01:55:27.260 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2024-04-24T01:55:26.935506+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":107980,"kb_used_data":660,"kb_used_omap":0,"kb_used_meta":107264,"kb_avail":377379380,"statfs":{"total":386547056640,"available":386436485120,"internally_reserved":0,"allocated":675840,"data_stored":1895372,"data_compressed":89889,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109838336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.779384"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-24T01:55:18.239215+0000","last_change":"2024-04-24T01:55:18.241117+0000","last_active":"2024-04-24T01:55:18.239215+0000","last_peered":"2024-04-24T01:55:18.239215+0000","last_clean":"2024-04-24T01:55:17.134477+0000","last_became_active":"2024-04-24T01:55:18.234313+0000","last_became_peered":"2024-04-24T01:55:18.234313+0000","last_unstale":"2024-04-24T01:55:18.239215+0000","last_undegraded":"2024-04-24T01:55:18.239215+0000","last_fullsized":"2024-04-24T01:55:18.239215+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-24T01:55:19.165132+0000","last_change":"2024-04-24T01:55:18.278351+0000","last_active":"2024-04-24T01:55:19.165132+0000","last_peered":"2024-04-24T01:55:19.165132+0000","last_clean":"2024-04-24T01:55:19.165132+0000","last_became_active":"2024-04-24T01:55:18.273668+0000","last_became_peered":"2024-04-24T01:55:18.273668+0000","last_unstale":"2024-04-24T01:55:19.165132+0000","last_undegraded":"2024-04-24T01:55:19.165132+0000","last_fullsized":"2024-04-24T01:55:19.165132+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-24T01:55:18.234008+0000","last_change":"2024-04-24T01:55:18.235763+0000","last_active":"2024-04-24T01:55:18.234008+0000","last_peered":"2024-04-24T01:55:18.234008+0000","last_clean":"2024-04-24T01:55:17.134477+0000","last_became_active":"2024-04-24T01:55:18.229175+0000","last_became_peered":"2024-04-24T01:55:18.229175+0000","last_unstale":"2024-04-24T01:55:18.234008+0000","last_undegraded":"2024-04-24T01:55:18.234008+0000","last_fullsized":"2024-04-24T01:55:18.234008+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-24T01:55:19.164444+0000","last_change":"2024-04-24T01:55:18.224238+0000","last_active":"2024-04-24T01:55:19.164444+0000","last_peered":"2024-04-24T01:55:19.164444+0000","last_clean":"2024-04-24T01:55:19.164444+0000","last_became_active":"2024-04-24T01:55:18.219131+0000","last_became_peered":"2024-04-24T01:55:18.219131+0000","last_unstale":"2024-04-24T01:55:19.164444+0000","last_undegraded":"2024-04-24T01:55:19.164444+0000","last_fullsized":"2024-04-24T01:55:19.164444+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-24T01:55:24.146605+0000","last_change":"2024-04-24T01:55:18.230893+0000","last_active":"2024-04-24T01:55:24.146605+0000","last_peered":"2024-04-24T01:55:24.146605+0000","last_clean":"2024-04-24T01:55:24.146605+0000","last_became_active":"2024-04-24T01:55:18.225188+0000","last_became_peered":"2024-04-24T01:55:18.225188+0000","last_unstale":"2024-04-24T01:55:24.146605+0000","last_undegraded":"2024-04-24T01:55:24.146605+0000","last_fullsized":"2024-04-24T01:55:24.146605+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-24T01:55:18.213174+0000","last_change":"2024-04-24T01:55:18.215251+0000","last_active":"2024-04-24T01:55:18.213174+0000","last_peered":"2024-04-24T01:55:18.213174+0000","last_clean":"2024-04-24T01:55:17.134477+0000","last_became_active":"2024-04-24T01:55:18.206605+0000","last_became_peered":"2024-04-24T01:55:18.206605+0000","last_unstale":"2024-04-24T01:55:18.213174+0000","last_undegraded":"2024-04-24T01:55:18.213174+0000","last_fullsized":"2024-04-24T01:55:18.213174+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-24T01:55:24.148481+0000","last_change":"2024-04-24T01:55:18.265905+0000","last_active":"2024-04-24T01:55:24.148481+0000","last_peered":"2024-04-24T01:55:24.148481+0000","last_clean":"2024-04-24T01:55:24.148481+0000","last_became_active":"2024-04-24T01:55:18.257020+0000","last_became_peered":"2024-04-24T01:55:18.257020+0000","last_unstale":"2024-04-24T01:55:24.148481+0000","last_undegraded":"2024-04-24T01:55:24.148481+0000","last_fullsized":"2024-04-24T01:55:24.148481+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-24T01:55:18.234590+0000","last_change":"2024-04-24T01:55:18.236148+0000","last_active":"2024-04-24T01:55:18.234590+0000","last_peered":"2024-04-24T01:55:18.234590+0000","last_clean":"2024-04-24T01:55:17.134477+0000","last_became_active":"2024-04-24T01:55:18.230154+0000","last_became_peered":"2024-04-24T01:55:18.230154+0000","last_unstale":"2024-04-24T01:55:18.234590+0000","last_undegraded":"2024-04-24T01:55:18.234590+0000","last_fullsized":"2024-04-24T01:55:18.234590+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'97","reported_seq":142,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-24T01:55:18.148759+0000","last_change":"2024-04-24T01:55:16.196470+0000","last_active":"2024-04-24T01:55:18.148759+0000","last_peered":"2024-04-24T01:55:18.148759+0000","last_clean":"2024-04-24T01:55:18.148759+0000","last_became_active":"2024-04-24T01:55:16.190464+0000","last_became_peered":"2024-04-24T01:55:16.190464+0000","last_unstale":"2024-04-24T01:55:18.148759+0000","last_undegraded":"2024-04-24T01:55:18.148759+0000","last_fullsized":"2024-04-24T01:55:18.148759+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:15.127115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:15.127115+0000","last_clean_scrub_stamp":"2024-04-24T01:55:15.127115+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":1}],"osd_stats":[{"osd":3,"up_from":10,"seq":42949672970,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27012,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344828,"statfs":{"total":96636764160,"available":96609103872,"internally_reserved":0,"allocated":180224,"data_stored":619830,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949672970,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27020,"kb_used_data":184,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344820,"statfs":{"total":96636764160,"available":96609095680,"internally_reserved":0,"allocated":188416,"data_stored":623040,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705675,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27008,"kb_used_data":184,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344832,"statfs":{"total":96636764160,"available":96609107968,"internally_reserved":0,"allocated":188416,"data_stored":623040,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26940,"kb_used_data":116,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344900,"statfs":{"total":96636764160,"available":96609177600,"internally_reserved":0,"allocated":118784,"data_stored":29462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T01:55:27.261 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-24T01:55:27.261 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump 2024-04-24T01:55:27.368 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:27.363+0000 7f6572d5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:27.372 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:27.371+0000 7f6572d5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:27.560 INFO:teuthology.orchestra.run.smithi079.stdout:epoch 15 2024-04-24T01:55:27.560 INFO:teuthology.orchestra.run.smithi079.stdout:fsid e600dd1c-0bef-4ea0-91c9-2f649f75622f 2024-04-24T01:55:27.561 INFO:teuthology.orchestra.run.smithi079.stdout:created 2024-04-24T01:55:06.017534+0000 2024-04-24T01:55:27.561 INFO:teuthology.orchestra.run.smithi079.stdout:modified 2024-04-24T01:55:19.141942+0000 2024-04-24T01:55:27.561 INFO:teuthology.orchestra.run.smithi079.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-04-24T01:55:27.561 INFO:teuthology.orchestra.run.smithi079.stdout:crush_version 4 2024-04-24T01:55:27.561 INFO:teuthology.orchestra.run.smithi079.stdout:full_ratio 0.95 2024-04-24T01:55:27.561 INFO:teuthology.orchestra.run.smithi079.stdout:backfillfull_ratio 0.9 2024-04-24T01:55:27.561 INFO:teuthology.orchestra.run.smithi079.stdout:nearfull_ratio 0.85 2024-04-24T01:55:27.561 INFO:teuthology.orchestra.run.smithi079.stdout:require_min_compat_client luminous 2024-04-24T01:55:27.561 INFO:teuthology.orchestra.run.smithi079.stdout:min_compat_client jewel 2024-04-24T01:55:27.561 INFO:teuthology.orchestra.run.smithi079.stdout:require_osd_release squid 2024-04-24T01:55:27.561 INFO:teuthology.orchestra.run.smithi079.stdout:stretch_mode_enabled false 2024-04-24T01:55:27.561 INFO:teuthology.orchestra.run.smithi079.stdout:allow_crimson=true 2024-04-24T01:55:27.561 INFO:teuthology.orchestra.run.smithi079.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 13 flags hashpspool,nopgchange,crimson stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 4.00 2024-04-24T01:55:27.562 INFO:teuthology.orchestra.run.smithi079.stdout:pool 2 'rbd' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 15 flags hashpspool,nopgchange,crimson stripe_width 0 application rbd read_balance_score 1.50 2024-04-24T01:55:27.562 INFO:teuthology.orchestra.run.smithi079.stdout:max_osd 4 2024-04-24T01:55:27.562 INFO:teuthology.orchestra.run.smithi079.stdout:osd.0 up in weight 1 up_from 9 up_thru 13 down_at 0 last_clean_interval [0,0) v2:172.21.15.79:6800/3830851766 v2:172.21.15.79:6803/3830851766 exists,up 1b20709f-f1e7-4d59-8402-0d13620f747e 2024-04-24T01:55:27.562 INFO:teuthology.orchestra.run.smithi079.stdout:osd.1 up in weight 1 up_from 9 up_thru 13 down_at 0 last_clean_interval [0,0) v2:172.21.15.79:6801/1823728464 v2:172.21.15.79:6802/1823728464 exists,up 54fd0eee-6d68-4c92-9ac7-6b21c87783f6 2024-04-24T01:55:27.562 INFO:teuthology.orchestra.run.smithi079.stdout:osd.2 up in weight 1 up_from 10 up_thru 13 down_at 0 last_clean_interval [0,0) v2:172.21.15.103:6803/2585849279 v2:172.21.15.103:6804/2585849279 exists,up 35820c96-5f2a-4e2e-bc4b-714ab8e1c841 2024-04-24T01:55:27.562 INFO:teuthology.orchestra.run.smithi079.stdout:osd.3 up in weight 1 up_from 10 up_thru 13 down_at 0 last_clean_interval [0,0) v2:172.21.15.103:6809/659350107 v2:172.21.15.103:6808/659350107 exists,up da975ea1-eeb6-4619-b73f-786b46dd8fe6 2024-04-24T01:55:27.573 INFO:tasks.ceph.ceph_manager.ceph:['osd.0 up in weight 1 up_from 9 up_thru 13 down_at 0 last_clean_interval [0,0) v2:172.21.15.79:6800/3830851766 v2:172.21.15.79:6803/3830851766 exists,up 1b20709f-f1e7-4d59-8402-0d13620f747e', 'osd.1 up in weight 1 up_from 9 up_thru 13 down_at 0 last_clean_interval [0,0) v2:172.21.15.79:6801/1823728464 v2:172.21.15.79:6802/1823728464 exists,up 54fd0eee-6d68-4c92-9ac7-6b21c87783f6', 'osd.2 up in weight 1 up_from 10 up_thru 13 down_at 0 last_clean_interval [0,0) v2:172.21.15.103:6803/2585849279 v2:172.21.15.103:6804/2585849279 exists,up 35820c96-5f2a-4e2e-bc4b-714ab8e1c841', 'osd.3 up in weight 1 up_from 10 up_thru 13 down_at 0 last_clean_interval [0,0) v2:172.21.15.103:6809/659350107 v2:172.21.15.103:6808/659350107 exists,up da975ea1-eeb6-4619-b73f-786b46dd8fe6'] 2024-04-24T01:55:27.574 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok config show 2024-04-24T01:55:27.679 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-04-24T01:55:27.679 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "mon.a", 2024-04-24T01:55:27.679 INFO:teuthology.orchestra.run.smithi079.stdout: "cluster": "ceph", 2024-04-24T01:55:27.679 INFO:teuthology.orchestra.run.smithi079.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-04-24T01:55:27.679 INFO:teuthology.orchestra.run.smithi079.stdout: "admin_socket_mode": "", 2024-04-24T01:55:27.679 INFO:teuthology.orchestra.run.smithi079.stdout: "auth_allow_insecure_global_id_reclaim": "true", 2024-04-24T01:55:27.679 INFO:teuthology.orchestra.run.smithi079.stdout: "auth_client_required": "cephx, none", 2024-04-24T01:55:27.679 INFO:teuthology.orchestra.run.smithi079.stdout: "auth_cluster_required": "cephx", 2024-04-24T01:55:27.680 INFO:teuthology.orchestra.run.smithi079.stdout: "auth_debug": "true", 2024-04-24T01:55:27.680 INFO:teuthology.orchestra.run.smithi079.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-04-24T01:55:27.680 INFO:teuthology.orchestra.run.smithi079.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-04-24T01:55:27.680 INFO:teuthology.orchestra.run.smithi079.stdout: "auth_service_required": "cephx", 2024-04-24T01:55:27.680 INFO:teuthology.orchestra.run.smithi079.stdout: "auth_service_ticket_ttl": "240.000000", 2024-04-24T01:55:27.680 INFO:teuthology.orchestra.run.smithi079.stdout: "auth_supported": "cephx", 2024-04-24T01:55:27.680 INFO:teuthology.orchestra.run.smithi079.stdout: "bdev_aio": "true", 2024-04-24T01:55:27.680 INFO:teuthology.orchestra.run.smithi079.stdout: "bdev_aio_max_queue_depth": "1024", 2024-04-24T01:55:27.680 INFO:teuthology.orchestra.run.smithi079.stdout: "bdev_aio_poll_ms": "250", 2024-04-24T01:55:27.680 INFO:teuthology.orchestra.run.smithi079.stdout: "bdev_aio_reap_max": "16", 2024-04-24T01:55:27.680 INFO:teuthology.orchestra.run.smithi079.stdout: "bdev_async_discard_threads": "0", 2024-04-24T01:55:27.680 INFO:teuthology.orchestra.run.smithi079.stdout: "bdev_block_size": "4096", 2024-04-24T01:55:27.680 INFO:teuthology.orchestra.run.smithi079.stdout: "bdev_debug_aio": "false", 2024-04-24T01:55:27.680 INFO:teuthology.orchestra.run.smithi079.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-04-24T01:55:27.681 INFO:teuthology.orchestra.run.smithi079.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-04-24T01:55:27.681 INFO:teuthology.orchestra.run.smithi079.stdout: "bdev_debug_inflight_ios": "false", 2024-04-24T01:55:27.681 INFO:teuthology.orchestra.run.smithi079.stdout: "bdev_enable_discard": "false", 2024-04-24T01:55:27.681 INFO:teuthology.orchestra.run.smithi079.stdout: "bdev_flock_retry": "3", 2024-04-24T01:55:27.681 INFO:teuthology.orchestra.run.smithi079.stdout: "bdev_flock_retry_interval": "0.100000", 2024-04-24T01:55:27.681 INFO:teuthology.orchestra.run.smithi079.stdout: "bdev_inject_crash": "0", 2024-04-24T01:55:27.681 INFO:teuthology.orchestra.run.smithi079.stdout: "bdev_inject_crash_flush_delay": "2", 2024-04-24T01:55:27.681 INFO:teuthology.orchestra.run.smithi079.stdout: "bdev_ioring": "false", 2024-04-24T01:55:27.681 INFO:teuthology.orchestra.run.smithi079.stdout: "bdev_ioring_hipri": "false", 2024-04-24T01:55:27.681 INFO:teuthology.orchestra.run.smithi079.stdout: "bdev_ioring_sqthread_poll": "false", 2024-04-24T01:55:27.681 INFO:teuthology.orchestra.run.smithi079.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-04-24T01:55:27.681 INFO:teuthology.orchestra.run.smithi079.stdout: "bdev_read_buffer_alignment": "4096", 2024-04-24T01:55:27.681 INFO:teuthology.orchestra.run.smithi079.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-04-24T01:55:27.682 INFO:teuthology.orchestra.run.smithi079.stdout: "bdev_type": "", 2024-04-24T01:55:27.682 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_alloc_size": "1048576", 2024-04-24T01:55:27.682 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_allocator": "hybrid", 2024-04-24T01:55:27.682 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_buffered_io": "true", 2024-04-24T01:55:27.682 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_check_for_zeros": "false", 2024-04-24T01:55:27.682 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_check_volume_selector_often": "false", 2024-04-24T01:55:27.682 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-04-24T01:55:27.682 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_compact_log_sync": "false", 2024-04-24T01:55:27.682 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-04-24T01:55:27.682 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-04-24T01:55:27.682 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_log_compact_min_size": "16777216", 2024-04-24T01:55:27.682 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_log_replay_check_allocations": "true", 2024-04-24T01:55:27.682 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_max_log_runway": "4194304", 2024-04-24T01:55:27.682 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_max_prefetch": "1048576", 2024-04-24T01:55:27.683 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_min_flush_size": "524288", 2024-04-24T01:55:27.683 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_min_log_runway": "1048576", 2024-04-24T01:55:27.683 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_replay_recovery": "false", 2024-04-24T01:55:27.683 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-04-24T01:55:27.683 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_shared_alloc_size": "65536", 2024-04-24T01:55:27.683 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_sync_write": "false", 2024-04-24T01:55:27.683 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-04-24T01:55:27.683 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-04-24T01:55:27.683 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-04-24T01:55:27.683 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_allocation_from_file": "true", 2024-04-24T01:55:27.683 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_allocator": "hybrid", 2024-04-24T01:55:27.683 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-04-24T01:55:27.683 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-04-24T01:55:27.683 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-04-24T01:55:27.683 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-04-24T01:55:27.684 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-04-24T01:55:27.684 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-04-24T01:55:27.684 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_blobid_prealloc": "10240", 2024-04-24T01:55:27.684 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_block_create": "true", 2024-04-24T01:55:27.684 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_block_db_create": "false", 2024-04-24T01:55:27.684 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_block_db_path": "", 2024-04-24T01:55:27.684 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_block_db_size": "0", 2024-04-24T01:55:27.684 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_block_path": "", 2024-04-24T01:55:27.684 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_block_preallocate_file": "false", 2024-04-24T01:55:27.684 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_block_size": "107374182400", 2024-04-24T01:55:27.684 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_block_wal_create": "false", 2024-04-24T01:55:27.684 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_block_wal_path": "", 2024-04-24T01:55:27.684 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_block_wal_size": "100663296", 2024-04-24T01:55:27.684 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_bluefs": "true", 2024-04-24T01:55:27.685 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-04-24T01:55:27.685 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_bluefs_env_mirror": "false", 2024-04-24T01:55:27.685 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-04-24T01:55:27.685 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-04-24T01:55:27.685 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-04-24T01:55:27.685 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-04-24T01:55:27.685 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-04-24T01:55:27.685 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-04-24T01:55:27.685 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_autotune": "true", 2024-04-24T01:55:27.685 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-04-24T01:55:27.685 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-04-24T01:55:27.685 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-04-24T01:55:27.685 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-04-24T01:55:27.685 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_size": "0", 2024-04-24T01:55:27.686 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-04-24T01:55:27.686 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-04-24T01:55:27.686 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-04-24T01:55:27.686 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-04-24T01:55:27.686 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_type": "2q", 2024-04-24T01:55:27.686 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-04-24T01:55:27.686 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_clone_cow": "true", 2024-04-24T01:55:27.686 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_compression_algorithm": "snappy", 2024-04-24T01:55:27.686 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_compression_max_blob_size": "0", 2024-04-24T01:55:27.686 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-04-24T01:55:27.686 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-04-24T01:55:27.686 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_compression_min_blob_size": "0", 2024-04-24T01:55:27.686 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-04-24T01:55:27.686 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-04-24T01:55:27.687 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_compression_mode": "none", 2024-04-24T01:55:27.687 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-04-24T01:55:27.687 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_csum_type": "crc32c", 2024-04-24T01:55:27.687 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_debug_enforce_min_alloc_size": "0", 2024-04-24T01:55:27.687 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_debug_enforce_settings": "default", 2024-04-24T01:55:27.687 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_debug_freelist": "false", 2024-04-24T01:55:27.687 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_debug_fsck_abort": "false", 2024-04-24T01:55:27.687 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-04-24T01:55:27.687 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-04-24T01:55:27.687 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_debug_inject_read_err": "false", 2024-04-24T01:55:27.687 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_debug_legacy_omap": "false", 2024-04-24T01:55:27.687 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-04-24T01:55:27.687 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-04-24T01:55:27.687 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-04-24T01:55:27.687 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-04-24T01:55:27.688 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_debug_prefill": "0.000000", 2024-04-24T01:55:27.688 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-04-24T01:55:27.688 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-04-24T01:55:27.688 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-04-24T01:55:27.688 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_debug_small_allocations": "0", 2024-04-24T01:55:27.688 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-04-24T01:55:27.688 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_default_buffered_read": "true", 2024-04-24T01:55:27.688 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_default_buffered_write": "false", 2024-04-24T01:55:27.688 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_deferred_batch_ops": "0", 2024-04-24T01:55:27.688 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-04-24T01:55:27.688 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-04-24T01:55:27.688 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_elastic_shared_blobs": "true", 2024-04-24T01:55:27.688 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-04-24T01:55:27.688 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-04-24T01:55:27.689 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-04-24T01:55:27.689 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-04-24T01:55:27.689 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-04-24T01:55:27.689 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_fail_eio": "false", 2024-04-24T01:55:27.689 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-04-24T01:55:27.689 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-04-24T01:55:27.689 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-04-24T01:55:27.689 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-04-24T01:55:27.689 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_fsck_on_mkfs": "true", 2024-04-24T01:55:27.689 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-04-24T01:55:27.689 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_fsck_on_mount": "false", 2024-04-24T01:55:27.689 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-04-24T01:55:27.689 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_fsck_on_umount": "false", 2024-04-24T01:55:27.689 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-04-24T01:55:27.689 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-04-24T01:55:27.690 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-04-24T01:55:27.690 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-04-24T01:55:27.690 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-04-24T01:55:27.690 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-04-24T01:55:27.690 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-04-24T01:55:27.690 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-04-24T01:55:27.690 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_ignore_data_csum": "false", 2024-04-24T01:55:27.691 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-04-24T01:55:27.691 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_kvbackend": "rocksdb", 2024-04-24T01:55:27.691 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-04-24T01:55:27.691 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-04-24T01:55:27.691 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_log_op_age": "5.000000", 2024-04-24T01:55:27.691 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_max_alloc_size": "0", 2024-04-24T01:55:27.691 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_max_blob_size": "0", 2024-04-24T01:55:27.691 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-04-24T01:55:27.692 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-04-24T01:55:27.692 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_max_defer_interval": "3.000000", 2024-04-24T01:55:27.692 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_max_deferred_txc": "32", 2024-04-24T01:55:27.692 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_min_alloc_size": "0", 2024-04-24T01:55:27.692 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-04-24T01:55:27.692 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-04-24T01:55:27.692 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_nid_prealloc": "1024", 2024-04-24T01:55:27.692 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_prefer_deferred_size": "0", 2024-04-24T01:55:27.692 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-04-24T01:55:27.692 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-04-24T01:55:27.692 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_qfsck_on_mount": "true", 2024-04-24T01:55:27.692 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_retry_disk_reads": "3", 2024-04-24T01:55:27.692 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_rocksdb_cf": "true", 2024-04-24T01:55:27.692 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-04-24T01:55:27.693 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-04-24T01:55:27.693 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_rocksdb_options_annex": "", 2024-04-24T01:55:27.693 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_spdk_coremask": "0x1", 2024-04-24T01:55:27.693 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_spdk_io_sleep": "5", 2024-04-24T01:55:27.693 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_spdk_max_io_completion": "0", 2024-04-24T01:55:27.693 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_spdk_mem": "512", 2024-04-24T01:55:27.693 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_sync_submit_transaction": "false", 2024-04-24T01:55:27.693 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_throttle_bytes": "67108864", 2024-04-24T01:55:27.693 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_throttle_cost_per_io": "0", 2024-04-24T01:55:27.693 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-04-24T01:55:27.693 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-04-24T01:55:27.693 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-04-24T01:55:27.693 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-04-24T01:55:27.693 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_tracing": "false", 2024-04-24T01:55:27.694 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-04-24T01:55:27.694 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-04-24T01:55:27.694 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_volume_selection_reserved": "0", 2024-04-24T01:55:27.694 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-04-24T01:55:27.694 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-04-24T01:55:27.694 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-04-24T01:55:27.694 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-04-24T01:55:27.694 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-04-24T01:55:27.694 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-04-24T01:55:27.694 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_zero_block_detection": "false", 2024-04-24T01:55:27.694 INFO:teuthology.orchestra.run.smithi079.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-04-24T01:55:27.694 INFO:teuthology.orchestra.run.smithi079.stdout: "cephfs_mirror_action_update_interval": "2", 2024-04-24T01:55:27.694 INFO:teuthology.orchestra.run.smithi079.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-04-24T01:55:27.694 INFO:teuthology.orchestra.run.smithi079.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-04-24T01:55:27.695 INFO:teuthology.orchestra.run.smithi079.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-04-24T01:55:27.695 INFO:teuthology.orchestra.run.smithi079.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-04-24T01:55:27.695 INFO:teuthology.orchestra.run.smithi079.stdout: "cephfs_mirror_mount_timeout": "10", 2024-04-24T01:55:27.695 INFO:teuthology.orchestra.run.smithi079.stdout: "cephfs_mirror_perf_stats_prio": "5", 2024-04-24T01:55:27.695 INFO:teuthology.orchestra.run.smithi079.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-04-24T01:55:27.695 INFO:teuthology.orchestra.run.smithi079.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-04-24T01:55:27.695 INFO:teuthology.orchestra.run.smithi079.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-04-24T01:55:27.695 INFO:teuthology.orchestra.run.smithi079.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-04-24T01:55:27.695 INFO:teuthology.orchestra.run.smithi079.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-04-24T01:55:27.695 INFO:teuthology.orchestra.run.smithi079.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-04-24T01:55:27.695 INFO:teuthology.orchestra.run.smithi079.stdout: "cephx_cluster_require_signatures": "false", 2024-04-24T01:55:27.695 INFO:teuthology.orchestra.run.smithi079.stdout: "cephx_cluster_require_version": "2", 2024-04-24T01:55:27.695 INFO:teuthology.orchestra.run.smithi079.stdout: "cephx_require_signatures": "false", 2024-04-24T01:55:27.695 INFO:teuthology.orchestra.run.smithi079.stdout: "cephx_require_version": "2", 2024-04-24T01:55:27.696 INFO:teuthology.orchestra.run.smithi079.stdout: "cephx_service_require_signatures": "false", 2024-04-24T01:55:27.696 INFO:teuthology.orchestra.run.smithi079.stdout: "cephx_service_require_version": "2", 2024-04-24T01:55:27.696 INFO:teuthology.orchestra.run.smithi079.stdout: "cephx_sign_messages": "true", 2024-04-24T01:55:27.696 INFO:teuthology.orchestra.run.smithi079.stdout: "chdir": "", 2024-04-24T01:55:27.696 INFO:teuthology.orchestra.run.smithi079.stdout: "client_acl_type": "", 2024-04-24T01:55:27.696 INFO:teuthology.orchestra.run.smithi079.stdout: "client_asio_thread_count": "2", 2024-04-24T01:55:27.696 INFO:teuthology.orchestra.run.smithi079.stdout: "client_cache_mid": "0.750000", 2024-04-24T01:55:27.696 INFO:teuthology.orchestra.run.smithi079.stdout: "client_cache_size": "16384", 2024-04-24T01:55:27.696 INFO:teuthology.orchestra.run.smithi079.stdout: "client_caps_release_delay": "5", 2024-04-24T01:55:27.696 INFO:teuthology.orchestra.run.smithi079.stdout: "client_check_pool_perm": "true", 2024-04-24T01:55:27.696 INFO:teuthology.orchestra.run.smithi079.stdout: "client_collect_and_send_global_metrics": "false", 2024-04-24T01:55:27.696 INFO:teuthology.orchestra.run.smithi079.stdout: "client_debug_force_sync_read": "false", 2024-04-24T01:55:27.696 INFO:teuthology.orchestra.run.smithi079.stdout: "client_debug_getattr_caps": "false", 2024-04-24T01:55:27.696 INFO:teuthology.orchestra.run.smithi079.stdout: "client_debug_inject_tick_delay": "0", 2024-04-24T01:55:27.696 INFO:teuthology.orchestra.run.smithi079.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-04-24T01:55:27.697 INFO:teuthology.orchestra.run.smithi079.stdout: "client_die_on_failed_remount": "false", 2024-04-24T01:55:27.697 INFO:teuthology.orchestra.run.smithi079.stdout: "client_dirsize_rbytes": "true", 2024-04-24T01:55:27.697 INFO:teuthology.orchestra.run.smithi079.stdout: "client_force_lazyio": "false", 2024-04-24T01:55:27.697 INFO:teuthology.orchestra.run.smithi079.stdout: "client_fs": "", 2024-04-24T01:55:27.697 INFO:teuthology.orchestra.run.smithi079.stdout: "client_inject_fixed_oldest_tid": "false", 2024-04-24T01:55:27.697 INFO:teuthology.orchestra.run.smithi079.stdout: "client_inject_release_failure": "false", 2024-04-24T01:55:27.697 INFO:teuthology.orchestra.run.smithi079.stdout: "client_max_inline_size": "4096", 2024-04-24T01:55:27.697 INFO:teuthology.orchestra.run.smithi079.stdout: "client_max_retries_on_remount_failure": "5", 2024-04-24T01:55:27.697 INFO:teuthology.orchestra.run.smithi079.stdout: "client_mds_namespace": "", 2024-04-24T01:55:27.697 INFO:teuthology.orchestra.run.smithi079.stdout: "client_metadata": "", 2024-04-24T01:55:27.697 INFO:teuthology.orchestra.run.smithi079.stdout: "client_mount_gid": "-1", 2024-04-24T01:55:27.697 INFO:teuthology.orchestra.run.smithi079.stdout: "client_mount_timeout": "300", 2024-04-24T01:55:27.697 INFO:teuthology.orchestra.run.smithi079.stdout: "client_mount_uid": "-1", 2024-04-24T01:55:27.698 INFO:teuthology.orchestra.run.smithi079.stdout: "client_mountpoint": "/", 2024-04-24T01:55:27.698 INFO:teuthology.orchestra.run.smithi079.stdout: "client_notify_timeout": "10", 2024-04-24T01:55:27.698 INFO:teuthology.orchestra.run.smithi079.stdout: "client_oc": "true", 2024-04-24T01:55:27.698 INFO:teuthology.orchestra.run.smithi079.stdout: "client_oc_max_dirty": "104857600", 2024-04-24T01:55:27.698 INFO:teuthology.orchestra.run.smithi079.stdout: "client_oc_max_dirty_age": "5.000000", 2024-04-24T01:55:27.698 INFO:teuthology.orchestra.run.smithi079.stdout: "client_oc_max_objects": "1000", 2024-04-24T01:55:27.698 INFO:teuthology.orchestra.run.smithi079.stdout: "client_oc_size": "209715200", 2024-04-24T01:55:27.698 INFO:teuthology.orchestra.run.smithi079.stdout: "client_oc_target_dirty": "8388608", 2024-04-24T01:55:27.698 INFO:teuthology.orchestra.run.smithi079.stdout: "client_permissions": "true", 2024-04-24T01:55:27.699 INFO:teuthology.orchestra.run.smithi079.stdout: "client_quota": "true", 2024-04-24T01:55:27.699 INFO:teuthology.orchestra.run.smithi079.stdout: "client_quota_df": "true", 2024-04-24T01:55:27.699 INFO:teuthology.orchestra.run.smithi079.stdout: "client_readahead_max_bytes": "0", 2024-04-24T01:55:27.699 INFO:teuthology.orchestra.run.smithi079.stdout: "client_readahead_max_periods": "4", 2024-04-24T01:55:27.699 INFO:teuthology.orchestra.run.smithi079.stdout: "client_readahead_min": "131072", 2024-04-24T01:55:27.699 INFO:teuthology.orchestra.run.smithi079.stdout: "client_reconnect_stale": "false", 2024-04-24T01:55:27.699 INFO:teuthology.orchestra.run.smithi079.stdout: "client_shutdown_timeout": "30", 2024-04-24T01:55:27.699 INFO:teuthology.orchestra.run.smithi079.stdout: "client_snapdir": ".snap", 2024-04-24T01:55:27.699 INFO:teuthology.orchestra.run.smithi079.stdout: "client_tick_interval": "1", 2024-04-24T01:55:27.699 INFO:teuthology.orchestra.run.smithi079.stdout: "client_trace": "", 2024-04-24T01:55:27.699 INFO:teuthology.orchestra.run.smithi079.stdout: "client_try_dentry_invalidate": "false", 2024-04-24T01:55:27.699 INFO:teuthology.orchestra.run.smithi079.stdout: "client_use_faked_inos": "false", 2024-04-24T01:55:27.699 INFO:teuthology.orchestra.run.smithi079.stdout: "client_use_random_mds": "false", 2024-04-24T01:55:27.699 INFO:teuthology.orchestra.run.smithi079.stdout: "clog_to_graylog": "false", 2024-04-24T01:55:27.699 INFO:teuthology.orchestra.run.smithi079.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-04-24T01:55:27.699 INFO:teuthology.orchestra.run.smithi079.stdout: "clog_to_graylog_port": "12201", 2024-04-24T01:55:27.700 INFO:teuthology.orchestra.run.smithi079.stdout: "clog_to_monitors": "default=true", 2024-04-24T01:55:27.700 INFO:teuthology.orchestra.run.smithi079.stdout: "clog_to_syslog": "false", 2024-04-24T01:55:27.700 INFO:teuthology.orchestra.run.smithi079.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-04-24T01:55:27.700 INFO:teuthology.orchestra.run.smithi079.stdout: "clog_to_syslog_level": "info", 2024-04-24T01:55:27.700 INFO:teuthology.orchestra.run.smithi079.stdout: "cluster_addr": "-", 2024-04-24T01:55:27.700 INFO:teuthology.orchestra.run.smithi079.stdout: "cluster_network": "", 2024-04-24T01:55:27.700 INFO:teuthology.orchestra.run.smithi079.stdout: "cluster_network_interface": "", 2024-04-24T01:55:27.700 INFO:teuthology.orchestra.run.smithi079.stdout: "compressor_zlib_isal": "false", 2024-04-24T01:55:27.700 INFO:teuthology.orchestra.run.smithi079.stdout: "compressor_zlib_level": "5", 2024-04-24T01:55:27.700 INFO:teuthology.orchestra.run.smithi079.stdout: "compressor_zlib_winsize": "-15", 2024-04-24T01:55:27.700 INFO:teuthology.orchestra.run.smithi079.stdout: "compressor_zstd_level": "1", 2024-04-24T01:55:27.700 INFO:teuthology.orchestra.run.smithi079.stdout: "container_image": "docker.io/ceph/daemon-base:latest-master-devel", 2024-04-24T01:55:27.700 INFO:teuthology.orchestra.run.smithi079.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-04-24T01:55:27.700 INFO:teuthology.orchestra.run.smithi079.stdout: "crimson_alien_op_num_threads": "6", 2024-04-24T01:55:27.700 INFO:teuthology.orchestra.run.smithi079.stdout: "crimson_alien_thread_cpu_cores": "", 2024-04-24T01:55:27.701 INFO:teuthology.orchestra.run.smithi079.stdout: "crimson_osd_obc_lru_size": "512", 2024-04-24T01:55:27.701 INFO:teuthology.orchestra.run.smithi079.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-04-24T01:55:27.701 INFO:teuthology.orchestra.run.smithi079.stdout: "crimson_osd_stat_interval": "0", 2024-04-24T01:55:27.701 INFO:teuthology.orchestra.run.smithi079.stdout: "crimson_seastar_cpu_cores": "", 2024-04-24T01:55:27.701 INFO:teuthology.orchestra.run.smithi079.stdout: "crush_location": "", 2024-04-24T01:55:27.701 INFO:teuthology.orchestra.run.smithi079.stdout: "crush_location_hook": "", 2024-04-24T01:55:27.701 INFO:teuthology.orchestra.run.smithi079.stdout: "crush_location_hook_timeout": "10", 2024-04-24T01:55:27.701 INFO:teuthology.orchestra.run.smithi079.stdout: "daemonize": "false", 2024-04-24T01:55:27.701 INFO:teuthology.orchestra.run.smithi079.stdout: "daos_pool": "tank", 2024-04-24T01:55:27.701 INFO:teuthology.orchestra.run.smithi079.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-04-24T01:55:27.701 INFO:teuthology.orchestra.run.smithi079.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-04-24T01:55:27.701 INFO:teuthology.orchestra.run.smithi079.stdout: "dbstore_db_name_prefix": "dbstore", 2024-04-24T01:55:27.701 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_alienstore": "0/5", 2024-04-24T01:55:27.701 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_allow_any_pool_priority": "false", 2024-04-24T01:55:27.701 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_asok": "1/5", 2024-04-24T01:55:27.702 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_asok_assert_abort": "false", 2024-04-24T01:55:27.702 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_asserts_on_shutdown": "true", 2024-04-24T01:55:27.702 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_auth": "20/20", 2024-04-24T01:55:27.702 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_bdev": "1/3", 2024-04-24T01:55:27.702 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_bluefs": "1/5", 2024-04-24T01:55:27.702 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_bluestore": "1/5", 2024-04-24T01:55:27.702 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_buffer": "0/1", 2024-04-24T01:55:27.702 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_ceph_exporter": "1/5", 2024-04-24T01:55:27.702 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_cephfs_mirror": "0/5", 2024-04-24T01:55:27.702 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_cephsqlite": "0/5", 2024-04-24T01:55:27.702 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_client": "0/5", 2024-04-24T01:55:27.702 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_compressor": "1/5", 2024-04-24T01:55:27.702 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_context": "0/1", 2024-04-24T01:55:27.702 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_crimson_interrupt": "0/5", 2024-04-24T01:55:27.703 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_crush": "1/1", 2024-04-24T01:55:27.703 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_crypto": "1/5", 2024-04-24T01:55:27.703 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_cyanstore": "0/5", 2024-04-24T01:55:27.703 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_deliberately_leak_memory": "false", 2024-04-24T01:55:27.703 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_disable_randomized_ping": "false", 2024-04-24T01:55:27.703 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_dpdk": "1/5", 2024-04-24T01:55:27.703 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_eventtrace": "1/5", 2024-04-24T01:55:27.703 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_filer": "0/1", 2024-04-24T01:55:27.703 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_filestore": "1/3", 2024-04-24T01:55:27.703 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_finisher": "1/1", 2024-04-24T01:55:27.703 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_fuse": "1/5", 2024-04-24T01:55:27.703 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_heartbeat_testing_span": "0", 2024-04-24T01:55:27.703 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_heartbeatmap": "1/5", 2024-04-24T01:55:27.703 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_immutable_obj_cache": "0/5", 2024-04-24T01:55:27.704 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_javaclient": "1/5", 2024-04-24T01:55:27.704 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_journal": "1/3", 2024-04-24T01:55:27.704 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_journaler": "0/5", 2024-04-24T01:55:27.704 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_kstore": "1/5", 2024-04-24T01:55:27.704 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_lockdep": "0/1", 2024-04-24T01:55:27.704 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_mclock": "1/5", 2024-04-24T01:55:27.704 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_mds": "1/5", 2024-04-24T01:55:27.704 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_mds_balancer": "1/5", 2024-04-24T01:55:27.704 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_mds_locker": "1/5", 2024-04-24T01:55:27.704 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_mds_log": "1/5", 2024-04-24T01:55:27.704 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_mds_log_expire": "1/5", 2024-04-24T01:55:27.704 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_mds_migrator": "1/5", 2024-04-24T01:55:27.704 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_mds_quiesce": "3/5", 2024-04-24T01:55:27.704 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_memstore": "1/5", 2024-04-24T01:55:27.704 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_mgr": "2/5", 2024-04-24T01:55:27.705 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_mgrc": "1/5", 2024-04-24T01:55:27.705 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_mon": "20/20", 2024-04-24T01:55:27.705 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_monc": "0/10", 2024-04-24T01:55:27.705 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_ms": "1/1", 2024-04-24T01:55:27.705 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_none": "0/5", 2024-04-24T01:55:27.705 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_objclass": "0/5", 2024-04-24T01:55:27.705 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_objectcacher": "0/5", 2024-04-24T01:55:27.705 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_objecter": "0/1", 2024-04-24T01:55:27.705 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_optracker": "0/5", 2024-04-24T01:55:27.705 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_osd": "1/5", 2024-04-24T01:55:27.705 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_paxos": "20/20", 2024-04-24T01:55:27.705 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_perfcounter": "1/5", 2024-04-24T01:55:27.705 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_prioritycache": "1/5", 2024-04-24T01:55:27.705 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_rados": "0/5", 2024-04-24T01:55:27.705 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_rbd": "0/5", 2024-04-24T01:55:27.706 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_rbd_mirror": "0/5", 2024-04-24T01:55:27.706 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_rbd_pwl": "0/5", 2024-04-24T01:55:27.706 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_rbd_replay": "0/5", 2024-04-24T01:55:27.706 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_refs": "0/0", 2024-04-24T01:55:27.706 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_reserver": "1/1", 2024-04-24T01:55:27.706 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_rgw": "1/5", 2024-04-24T01:55:27.706 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_rgw_access": "1/5", 2024-04-24T01:55:27.706 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_rgw_datacache": "1/5", 2024-04-24T01:55:27.706 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_rgw_dbstore": "1/5", 2024-04-24T01:55:27.706 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_rgw_flight": "1/5", 2024-04-24T01:55:27.706 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_rgw_lifecycle": "1/5", 2024-04-24T01:55:27.706 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_rgw_notification": "1/5", 2024-04-24T01:55:27.706 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_rgw_sync": "1/5", 2024-04-24T01:55:27.706 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_rocksdb": "4/5", 2024-04-24T01:55:27.706 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_seastore": "0/5", 2024-04-24T01:55:27.707 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_seastore_backref": "0/5", 2024-04-24T01:55:27.707 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_seastore_cache": "0/5", 2024-04-24T01:55:27.707 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_seastore_cleaner": "0/5", 2024-04-24T01:55:27.707 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_seastore_device": "0/5", 2024-04-24T01:55:27.707 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_seastore_epm": "0/5", 2024-04-24T01:55:27.707 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-04-24T01:55:27.707 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_seastore_journal": "0/5", 2024-04-24T01:55:27.707 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_seastore_lba": "0/5", 2024-04-24T01:55:27.707 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_seastore_odata": "0/5", 2024-04-24T01:55:27.707 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_seastore_omap": "0/5", 2024-04-24T01:55:27.707 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_seastore_onode": "0/5", 2024-04-24T01:55:27.707 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_seastore_t": "0/5", 2024-04-24T01:55:27.707 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_seastore_tm": "0/5", 2024-04-24T01:55:27.707 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_striper": "0/1", 2024-04-24T01:55:27.708 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_test": "0/5", 2024-04-24T01:55:27.708 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_throttle": "1/1", 2024-04-24T01:55:27.708 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_timer": "0/1", 2024-04-24T01:55:27.708 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_tp": "0/5", 2024-04-24T01:55:27.708 INFO:teuthology.orchestra.run.smithi079.stdout: "debug_trace": "1/5", 2024-04-24T01:55:27.708 INFO:teuthology.orchestra.run.smithi079.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-04-24T01:55:27.708 INFO:teuthology.orchestra.run.smithi079.stdout: "device_failure_prediction_mode": "none", 2024-04-24T01:55:27.708 INFO:teuthology.orchestra.run.smithi079.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "crimson", 2024-04-24T01:55:27.708 INFO:teuthology.orchestra.run.smithi079.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-04-24T01:55:27.708 INFO:teuthology.orchestra.run.smithi079.stdout: "err_to_graylog": "false", 2024-04-24T01:55:27.708 INFO:teuthology.orchestra.run.smithi079.stdout: "err_to_journald": "false", 2024-04-24T01:55:27.708 INFO:teuthology.orchestra.run.smithi079.stdout: "err_to_stderr": "true", 2024-04-24T01:55:27.708 INFO:teuthology.orchestra.run.smithi079.stdout: "err_to_syslog": "false", 2024-04-24T01:55:27.708 INFO:teuthology.orchestra.run.smithi079.stdout: "event_tracing": "false", 2024-04-24T01:55:27.709 INFO:teuthology.orchestra.run.smithi079.stdout: "exporter_addr": "0.0.0.0", 2024-04-24T01:55:27.709 INFO:teuthology.orchestra.run.smithi079.stdout: "exporter_http_port": "9926", 2024-04-24T01:55:27.709 INFO:teuthology.orchestra.run.smithi079.stdout: "exporter_prio_limit": "5", 2024-04-24T01:55:27.709 INFO:teuthology.orchestra.run.smithi079.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-04-24T01:55:27.709 INFO:teuthology.orchestra.run.smithi079.stdout: "exporter_sort_metrics": "true", 2024-04-24T01:55:27.709 INFO:teuthology.orchestra.run.smithi079.stdout: "exporter_stats_period": "5", 2024-04-24T01:55:27.709 INFO:teuthology.orchestra.run.smithi079.stdout: "fake_statfs_for_testing": "0", 2024-04-24T01:55:27.709 INFO:teuthology.orchestra.run.smithi079.stdout: "fatal_signal_handlers": "true", 2024-04-24T01:55:27.709 INFO:teuthology.orchestra.run.smithi079.stdout: "filer_max_purge_ops": "10", 2024-04-24T01:55:27.709 INFO:teuthology.orchestra.run.smithi079.stdout: "filer_max_truncate_ops": "128", 2024-04-24T01:55:27.709 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_apply_finisher_threads": "1", 2024-04-24T01:55:27.709 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_blackhole": "false", 2024-04-24T01:55:27.709 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_btrfs_clone_range": "true", 2024-04-24T01:55:27.709 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_btrfs_snap": "true", 2024-04-24T01:55:27.709 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_caller_concurrency": "10", 2024-04-24T01:55:27.710 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_collect_device_partition_information": "true", 2024-04-24T01:55:27.710 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_commit_timeout": "600.000000", 2024-04-24T01:55:27.710 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_debug_inject_read_err": "false", 2024-04-24T01:55:27.710 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_debug_omap_check": "false", 2024-04-24T01:55:27.710 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_debug_verify_split": "false", 2024-04-24T01:55:27.710 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_dump_file": "", 2024-04-24T01:55:27.710 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-04-24T01:55:27.710 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-04-24T01:55:27.710 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_fadvise": "true", 2024-04-24T01:55:27.710 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_fail_eio": "true", 2024-04-24T01:55:27.710 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_fd_cache_shards": "16", 2024-04-24T01:55:27.710 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_fd_cache_size": "128", 2024-04-24T01:55:27.710 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_fiemap": "false", 2024-04-24T01:55:27.711 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_fiemap_threshold": "4096", 2024-04-24T01:55:27.711 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-04-24T01:55:27.711 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_index_retry_probability": "0.000000", 2024-04-24T01:55:27.711 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_inject_stall": "0", 2024-04-24T01:55:27.711 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_journal_parallel": "false", 2024-04-24T01:55:27.711 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_journal_trailing": "false", 2024-04-24T01:55:27.711 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_journal_writeahead": "false", 2024-04-24T01:55:27.711 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_kill_at": "0", 2024-04-24T01:55:27.711 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-04-24T01:55:27.711 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_max_inline_xattr_size": "0", 2024-04-24T01:55:27.711 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-04-24T01:55:27.711 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-04-24T01:55:27.711 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-04-24T01:55:27.711 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_max_inline_xattrs": "0", 2024-04-24T01:55:27.711 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-04-24T01:55:27.712 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_max_inline_xattrs_other": "2", 2024-04-24T01:55:27.712 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-04-24T01:55:27.712 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_max_sync_interval": "5.000000", 2024-04-24T01:55:27.712 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_max_xattr_value_size": "0", 2024-04-24T01:55:27.712 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-04-24T01:55:27.712 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-04-24T01:55:27.712 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-04-24T01:55:27.712 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_merge_threshold": "-10", 2024-04-24T01:55:27.712 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_min_sync_interval": "0.010000", 2024-04-24T01:55:27.712 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_odsync_write": "false", 2024-04-24T01:55:27.712 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_omap_backend": "rocksdb", 2024-04-24T01:55:27.712 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_omap_backend_path": "", 2024-04-24T01:55:27.712 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_omap_header_cache_size": "1024", 2024-04-24T01:55:27.713 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_ondisk_finisher_threads": "1", 2024-04-24T01:55:27.713 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-04-24T01:55:27.713 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_op_thread_timeout": "60", 2024-04-24T01:55:27.713 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_op_threads": "2", 2024-04-24T01:55:27.713 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_punch_hole": "false", 2024-04-24T01:55:27.713 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-04-24T01:55:27.713 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-04-24T01:55:27.713 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-04-24T01:55:27.713 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-04-24T01:55:27.713 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-04-24T01:55:27.713 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_queue_max_bytes": "104857600", 2024-04-24T01:55:27.713 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-04-24T01:55:27.713 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-04-24T01:55:27.713 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-04-24T01:55:27.714 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_queue_max_ops": "50", 2024-04-24T01:55:27.714 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-04-24T01:55:27.714 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_seek_data_hole": "false", 2024-04-24T01:55:27.714 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_sloppy_crc": "false", 2024-04-24T01:55:27.714 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-04-24T01:55:27.714 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_splice": "false", 2024-04-24T01:55:27.714 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_split_multiple": "2", 2024-04-24T01:55:27.714 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_split_rand_factor": "20", 2024-04-24T01:55:27.714 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_update_to": "1000", 2024-04-24T01:55:27.714 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-04-24T01:55:27.714 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-04-24T01:55:27.714 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-04-24T01:55:27.714 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-04-24T01:55:27.714 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-04-24T01:55:27.714 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-04-24T01:55:27.715 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_wbthrottle_enable": "true", 2024-04-24T01:55:27.715 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-04-24T01:55:27.715 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-04-24T01:55:27.715 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-04-24T01:55:27.715 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-04-24T01:55:27.715 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-04-24T01:55:27.715 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-04-24T01:55:27.715 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_xfs_extsize": "false", 2024-04-24T01:55:27.715 INFO:teuthology.orchestra.run.smithi079.stdout: "filestore_zfs_snap": "false", 2024-04-24T01:55:27.715 INFO:teuthology.orchestra.run.smithi079.stdout: "fio_dir": "/tmp/fio", 2024-04-24T01:55:27.715 INFO:teuthology.orchestra.run.smithi079.stdout: "fsid": "e600dd1c-0bef-4ea0-91c9-2f649f75622f", 2024-04-24T01:55:27.715 INFO:teuthology.orchestra.run.smithi079.stdout: "fuse_allow_other": "true", 2024-04-24T01:55:27.715 INFO:teuthology.orchestra.run.smithi079.stdout: "fuse_atomic_o_trunc": "true", 2024-04-24T01:55:27.715 INFO:teuthology.orchestra.run.smithi079.stdout: "fuse_big_writes": "true", 2024-04-24T01:55:27.715 INFO:teuthology.orchestra.run.smithi079.stdout: "fuse_debug": "false", 2024-04-24T01:55:27.716 INFO:teuthology.orchestra.run.smithi079.stdout: "fuse_default_permissions": "false", 2024-04-24T01:55:27.716 INFO:teuthology.orchestra.run.smithi079.stdout: "fuse_disable_pagecache": "false", 2024-04-24T01:55:27.716 INFO:teuthology.orchestra.run.smithi079.stdout: "fuse_max_write": "0", 2024-04-24T01:55:27.716 INFO:teuthology.orchestra.run.smithi079.stdout: "fuse_multithreaded": "true", 2024-04-24T01:55:27.716 INFO:teuthology.orchestra.run.smithi079.stdout: "fuse_require_active_mds": "true", 2024-04-24T01:55:27.716 INFO:teuthology.orchestra.run.smithi079.stdout: "fuse_set_user_groups": "true", 2024-04-24T01:55:27.716 INFO:teuthology.orchestra.run.smithi079.stdout: "fuse_splice_move": "true", 2024-04-24T01:55:27.716 INFO:teuthology.orchestra.run.smithi079.stdout: "fuse_splice_read": "true", 2024-04-24T01:55:27.716 INFO:teuthology.orchestra.run.smithi079.stdout: "fuse_splice_write": "true", 2024-04-24T01:55:27.716 INFO:teuthology.orchestra.run.smithi079.stdout: "fuse_syncfs_on_mksnap": "true", 2024-04-24T01:55:27.716 INFO:teuthology.orchestra.run.smithi079.stdout: "fuse_use_invalidate_cb": "true", 2024-04-24T01:55:27.716 INFO:teuthology.orchestra.run.smithi079.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-04-24T01:55:27.716 INFO:teuthology.orchestra.run.smithi079.stdout: "gss_target_name": "ceph", 2024-04-24T01:55:27.716 INFO:teuthology.orchestra.run.smithi079.stdout: "heartbeat_file": "", 2024-04-24T01:55:27.717 INFO:teuthology.orchestra.run.smithi079.stdout: "heartbeat_inject_failure": "0", 2024-04-24T01:55:27.717 INFO:teuthology.orchestra.run.smithi079.stdout: "heartbeat_interval": "5", 2024-04-24T01:55:27.717 INFO:teuthology.orchestra.run.smithi079.stdout: "host": "smithi079", 2024-04-24T01:55:27.717 INFO:teuthology.orchestra.run.smithi079.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-04-24T01:55:27.717 INFO:teuthology.orchestra.run.smithi079.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-04-24T01:55:27.717 INFO:teuthology.orchestra.run.smithi079.stdout: "immutable_object_cache_max_size": "1073741824", 2024-04-24T01:55:27.717 INFO:teuthology.orchestra.run.smithi079.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-04-24T01:55:27.717 INFO:teuthology.orchestra.run.smithi079.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-04-24T01:55:27.717 INFO:teuthology.orchestra.run.smithi079.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-04-24T01:55:27.717 INFO:teuthology.orchestra.run.smithi079.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-04-24T01:55:27.717 INFO:teuthology.orchestra.run.smithi079.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-04-24T01:55:27.717 INFO:teuthology.orchestra.run.smithi079.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-04-24T01:55:27.717 INFO:teuthology.orchestra.run.smithi079.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-04-24T01:55:27.717 INFO:teuthology.orchestra.run.smithi079.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-04-24T01:55:27.718 INFO:teuthology.orchestra.run.smithi079.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-04-24T01:55:27.718 INFO:teuthology.orchestra.run.smithi079.stdout: "immutable_object_cache_watermark": "0.900000", 2024-04-24T01:55:27.718 INFO:teuthology.orchestra.run.smithi079.stdout: "inject_early_sigterm": "false", 2024-04-24T01:55:27.718 INFO:teuthology.orchestra.run.smithi079.stdout: "jaeger_agent_port": "6799", 2024-04-24T01:55:27.718 INFO:teuthology.orchestra.run.smithi079.stdout: "jaeger_tracing_enable": "false", 2024-04-24T01:55:27.718 INFO:teuthology.orchestra.run.smithi079.stdout: "journal_aio": "true", 2024-04-24T01:55:27.718 INFO:teuthology.orchestra.run.smithi079.stdout: "journal_align_min_size": "65536", 2024-04-24T01:55:27.718 INFO:teuthology.orchestra.run.smithi079.stdout: "journal_block_align": "true", 2024-04-24T01:55:27.718 INFO:teuthology.orchestra.run.smithi079.stdout: "journal_block_size": "4096", 2024-04-24T01:55:27.718 INFO:teuthology.orchestra.run.smithi079.stdout: "journal_dio": "true", 2024-04-24T01:55:27.718 INFO:teuthology.orchestra.run.smithi079.stdout: "journal_discard": "false", 2024-04-24T01:55:27.718 INFO:teuthology.orchestra.run.smithi079.stdout: "journal_force_aio": "false", 2024-04-24T01:55:27.718 INFO:teuthology.orchestra.run.smithi079.stdout: "journal_ignore_corruption": "false", 2024-04-24T01:55:27.718 INFO:teuthology.orchestra.run.smithi079.stdout: "journal_max_write_bytes": "10485760", 2024-04-24T01:55:27.718 INFO:teuthology.orchestra.run.smithi079.stdout: "journal_max_write_entries": "100", 2024-04-24T01:55:27.719 INFO:teuthology.orchestra.run.smithi079.stdout: "journal_replay_from": "0", 2024-04-24T01:55:27.719 INFO:teuthology.orchestra.run.smithi079.stdout: "journal_throttle_high_multiple": "0.000000", 2024-04-24T01:55:27.719 INFO:teuthology.orchestra.run.smithi079.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-04-24T01:55:27.719 INFO:teuthology.orchestra.run.smithi079.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-04-24T01:55:27.719 INFO:teuthology.orchestra.run.smithi079.stdout: "journal_throttle_max_multiple": "0.000000", 2024-04-24T01:55:27.719 INFO:teuthology.orchestra.run.smithi079.stdout: "journal_write_header_frequency": "0", 2024-04-24T01:55:27.719 INFO:teuthology.orchestra.run.smithi079.stdout: "journal_zero_on_create": "false", 2024-04-24T01:55:27.719 INFO:teuthology.orchestra.run.smithi079.stdout: "journaler_prefetch_periods": "10", 2024-04-24T01:55:27.719 INFO:teuthology.orchestra.run.smithi079.stdout: "journaler_prezero_periods": "5", 2024-04-24T01:55:27.719 INFO:teuthology.orchestra.run.smithi079.stdout: "journaler_write_head_interval": "15", 2024-04-24T01:55:27.719 INFO:teuthology.orchestra.run.smithi079.stdout: "key": "", 2024-04-24T01:55:27.719 INFO:teuthology.orchestra.run.smithi079.stdout: "keyfile": "", 2024-04-24T01:55:27.719 INFO:teuthology.orchestra.run.smithi079.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-04-24T01:55:27.719 INFO:teuthology.orchestra.run.smithi079.stdout: "kstore_backend": "rocksdb", 2024-04-24T01:55:27.720 INFO:teuthology.orchestra.run.smithi079.stdout: "kstore_default_stripe_size": "65536", 2024-04-24T01:55:27.720 INFO:teuthology.orchestra.run.smithi079.stdout: "kstore_fsck_on_mount": "false", 2024-04-24T01:55:27.720 INFO:teuthology.orchestra.run.smithi079.stdout: "kstore_fsck_on_mount_deep": "true", 2024-04-24T01:55:27.720 INFO:teuthology.orchestra.run.smithi079.stdout: "kstore_max_bytes": "67108864", 2024-04-24T01:55:27.720 INFO:teuthology.orchestra.run.smithi079.stdout: "kstore_max_ops": "512", 2024-04-24T01:55:27.720 INFO:teuthology.orchestra.run.smithi079.stdout: "kstore_nid_prealloc": "1024", 2024-04-24T01:55:27.720 INFO:teuthology.orchestra.run.smithi079.stdout: "kstore_onode_map_size": "1024", 2024-04-24T01:55:27.720 INFO:teuthology.orchestra.run.smithi079.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-04-24T01:55:27.720 INFO:teuthology.orchestra.run.smithi079.stdout: "kstore_sync_submit_transaction": "false", 2024-04-24T01:55:27.720 INFO:teuthology.orchestra.run.smithi079.stdout: "kstore_sync_transaction": "false", 2024-04-24T01:55:27.720 INFO:teuthology.orchestra.run.smithi079.stdout: "librados_thread_count": "2", 2024-04-24T01:55:27.720 INFO:teuthology.orchestra.run.smithi079.stdout: "lockdep": "false", 2024-04-24T01:55:27.720 INFO:teuthology.orchestra.run.smithi079.stdout: "lockdep_force_backtrace": "false", 2024-04-24T01:55:27.720 INFO:teuthology.orchestra.run.smithi079.stdout: "log_coarse_timestamps": "true", 2024-04-24T01:55:27.720 INFO:teuthology.orchestra.run.smithi079.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-04-24T01:55:27.721 INFO:teuthology.orchestra.run.smithi079.stdout: "log_flush_on_exit": "false", 2024-04-24T01:55:27.721 INFO:teuthology.orchestra.run.smithi079.stdout: "log_graylog_host": "127.0.0.1", 2024-04-24T01:55:27.721 INFO:teuthology.orchestra.run.smithi079.stdout: "log_graylog_port": "12201", 2024-04-24T01:55:27.721 INFO:teuthology.orchestra.run.smithi079.stdout: "log_max_new": "1000", 2024-04-24T01:55:27.721 INFO:teuthology.orchestra.run.smithi079.stdout: "log_max_recent": "10000", 2024-04-24T01:55:27.721 INFO:teuthology.orchestra.run.smithi079.stdout: "log_stderr_prefix": "", 2024-04-24T01:55:27.721 INFO:teuthology.orchestra.run.smithi079.stdout: "log_stop_at_utilization": "0.970000", 2024-04-24T01:55:27.721 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_file": "true", 2024-04-24T01:55:27.721 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_graylog": "false", 2024-04-24T01:55:27.721 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_journald": "false", 2024-04-24T01:55:27.721 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_stderr": "false", 2024-04-24T01:55:27.721 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_syslog": "false", 2024-04-24T01:55:27.721 INFO:teuthology.orchestra.run.smithi079.stdout: "max_rotating_auth_attempts": "10", 2024-04-24T01:55:27.721 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-04-24T01:55:27.722 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_action_on_write_error": "1", 2024-04-24T01:55:27.722 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_alternate_name_max": "8192", 2024-04-24T01:55:27.722 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_asio_thread_count": "2", 2024-04-24T01:55:27.722 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_bal_export_pin": "true", 2024-04-24T01:55:27.722 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_bal_fragment_dirs": "true", 2024-04-24T01:55:27.722 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-04-24T01:55:27.722 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_bal_fragment_interval": "5", 2024-04-24T01:55:27.722 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_bal_fragment_size_max": "100000", 2024-04-24T01:55:27.722 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_bal_idle_threshold": "0.000000", 2024-04-24T01:55:27.722 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_bal_interval": "10", 2024-04-24T01:55:27.722 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_bal_max": "-1", 2024-04-24T01:55:27.722 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_bal_max_until": "-1", 2024-04-24T01:55:27.722 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_bal_merge_size": "50", 2024-04-24T01:55:27.722 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_bal_midchunk": "0.300000", 2024-04-24T01:55:27.723 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_bal_min_rebalance": "0.100000", 2024-04-24T01:55:27.723 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_bal_min_start": "0.200000", 2024-04-24T01:55:27.723 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_bal_minchunk": "0.001000", 2024-04-24T01:55:27.723 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_bal_mode": "0", 2024-04-24T01:55:27.723 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_bal_need_max": "1.200000", 2024-04-24T01:55:27.723 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_bal_need_min": "0.800000", 2024-04-24T01:55:27.723 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_bal_overload_epochs": "2", 2024-04-24T01:55:27.723 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-04-24T01:55:27.723 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_bal_sample_interval": "3.000000", 2024-04-24T01:55:27.723 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_bal_split_bits": "3", 2024-04-24T01:55:27.723 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_bal_split_rd": "25000.000000", 2024-04-24T01:55:27.723 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_bal_split_size": "10000", 2024-04-24T01:55:27.723 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_bal_split_wr": "10000.000000", 2024-04-24T01:55:27.723 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_bal_target_decay": "10.000000", 2024-04-24T01:55:27.724 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-04-24T01:55:27.724 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_beacon_grace": "15.000000", 2024-04-24T01:55:27.724 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_beacon_interval": "4.000000", 2024-04-24T01:55:27.724 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_beacon_mon_down_grace": "60", 2024-04-24T01:55:27.724 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_cache_memory_limit": "4294967296", 2024-04-24T01:55:27.724 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_cache_mid": "0.700000", 2024-04-24T01:55:27.724 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_cache_quiesce_decay_rate": "1.000000", 2024-04-24T01:55:27.724 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_cache_quiesce_delay": "0", 2024-04-24T01:55:27.724 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_cache_quiesce_sleep": "200", 2024-04-24T01:55:27.724 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_cache_quiesce_splitauth": "true", 2024-04-24T01:55:27.724 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_cache_quiesce_threshold": "524288", 2024-04-24T01:55:27.724 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_cache_release_free_interval": "10", 2024-04-24T01:55:27.724 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_cache_reservation": "0.050000", 2024-04-24T01:55:27.724 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-04-24T01:55:27.724 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_cache_trim_interval": "1", 2024-04-24T01:55:27.725 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_cache_trim_threshold": "262144", 2024-04-24T01:55:27.725 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-04-24T01:55:27.725 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-04-24T01:55:27.725 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_client_delegate_inos_pct": "50", 2024-04-24T01:55:27.725 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_client_prealloc_inos": "1000", 2024-04-24T01:55:27.725 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-04-24T01:55:27.725 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_connect_bootstrapping": "false", 2024-04-24T01:55:27.725 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_damage_table_max_entries": "10000", 2024-04-24T01:55:27.725 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-04-24T01:55:27.725 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_debug_auth_pins": "false", 2024-04-24T01:55:27.725 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_debug_frag": "false", 2024-04-24T01:55:27.725 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_debug_scatterstat": "false", 2024-04-24T01:55:27.725 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_debug_subtrees": "false", 2024-04-24T01:55:27.725 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_decay_halflife": "5.000000", 2024-04-24T01:55:27.726 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_default_dir_hash": "2", 2024-04-24T01:55:27.726 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_defer_session_stale": "true", 2024-04-24T01:55:27.726 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_deny_all_reconnect": "false", 2024-04-24T01:55:27.726 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_dir_keys_per_op": "16384", 2024-04-24T01:55:27.726 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_dir_max_commit_size": "10", 2024-04-24T01:55:27.726 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_dir_max_entries": "0", 2024-04-24T01:55:27.726 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_dir_prefetch": "true", 2024-04-24T01:55:27.726 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_dirstat_min_interval": "1.000000", 2024-04-24T01:55:27.726 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_dump_cache_after_rejoin": "false", 2024-04-24T01:55:27.726 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_dump_cache_on_map": "false", 2024-04-24T01:55:27.726 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_dump_cache_threshold_file": "0", 2024-04-24T01:55:27.726 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-04-24T01:55:27.726 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_early_reply": "true", 2024-04-24T01:55:27.726 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_enable_op_tracker": "true", 2024-04-24T01:55:27.726 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_enforce_unique_name": "true", 2024-04-24T01:55:27.727 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_export_ephemeral_distributed": "true", 2024-04-24T01:55:27.727 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-04-24T01:55:27.727 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_export_ephemeral_random": "true", 2024-04-24T01:55:27.727 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-04-24T01:55:27.727 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-04-24T01:55:27.727 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_forward_all_requests_to_auth": "false", 2024-04-24T01:55:27.727 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-04-24T01:55:27.727 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-04-24T01:55:27.727 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-04-24T01:55:27.727 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-04-24T01:55:27.727 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_health_cache_threshold": "1.500000", 2024-04-24T01:55:27.727 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_health_summarize_threshold": "10", 2024-04-24T01:55:27.727 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_heartbeat_grace": "15.000000", 2024-04-24T01:55:27.727 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_heartbeat_reset_grace": "1000", 2024-04-24T01:55:27.728 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_inject_health_dummy": "false", 2024-04-24T01:55:27.728 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-04-24T01:55:27.728 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_inject_migrator_session_race": "false", 2024-04-24T01:55:27.728 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-04-24T01:55:27.728 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-04-24T01:55:27.728 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-04-24T01:55:27.728 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_join_fs": "", 2024-04-24T01:55:27.728 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_journal_format": "1", 2024-04-24T01:55:27.728 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_kill_create_at": "0", 2024-04-24T01:55:27.728 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_kill_export_at": "0", 2024-04-24T01:55:27.728 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_kill_import_at": "0", 2024-04-24T01:55:27.728 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_kill_journal_at": "0", 2024-04-24T01:55:27.728 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_kill_journal_expire_at": "0", 2024-04-24T01:55:27.728 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_kill_journal_replay_at": "0", 2024-04-24T01:55:27.728 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_kill_link_at": "0", 2024-04-24T01:55:27.729 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_kill_mdstable_at": "0", 2024-04-24T01:55:27.729 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_kill_openc_at": "0", 2024-04-24T01:55:27.729 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_kill_rename_at": "0", 2024-04-24T01:55:27.729 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_kill_shutdown_at": "0", 2024-04-24T01:55:27.729 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_kill_skip_replaying_inotable": "false", 2024-04-24T01:55:27.729 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_log_event_large_threshold": "524288", 2024-04-24T01:55:27.729 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_log_events_per_segment": "1024", 2024-04-24T01:55:27.729 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_log_major_segment_event_ratio": "12", 2024-04-24T01:55:27.729 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_log_max_events": "-1", 2024-04-24T01:55:27.729 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_log_max_segments": "128", 2024-04-24T01:55:27.729 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_log_pause": "false", 2024-04-24T01:55:27.729 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_log_segment_size": "0", 2024-04-24T01:55:27.729 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_log_skip_corrupt_events": "false", 2024-04-24T01:55:27.729 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_log_skip_unbounded_events": "false", 2024-04-24T01:55:27.729 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-04-24T01:55:27.730 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_log_trim_threshold": "128", 2024-04-24T01:55:27.730 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-04-24T01:55:27.730 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_log_warn_factor": "2.000000", 2024-04-24T01:55:27.730 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_max_caps_per_client": "1048576", 2024-04-24T01:55:27.730 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_max_completed_flushes": "100000", 2024-04-24T01:55:27.730 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_max_completed_requests": "100000", 2024-04-24T01:55:27.730 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_max_export_size": "20971520", 2024-04-24T01:55:27.730 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_max_file_recover": "32", 2024-04-24T01:55:27.730 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_max_purge_files": "64", 2024-04-24T01:55:27.730 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_max_purge_ops": "8192", 2024-04-24T01:55:27.730 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-04-24T01:55:27.730 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-04-24T01:55:27.730 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_max_snaps_per_dir": "100", 2024-04-24T01:55:27.730 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_metrics_update_interval": "2", 2024-04-24T01:55:27.730 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_min_caps_per_client": "100", 2024-04-24T01:55:27.731 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_min_caps_working_set": "10000", 2024-04-24T01:55:27.731 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-04-24T01:55:27.731 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_numa_node": "-1", 2024-04-24T01:55:27.731 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-04-24T01:55:27.731 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_op_complaint_time": "30.000000", 2024-04-24T01:55:27.731 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_op_history_duration": "600", 2024-04-24T01:55:27.731 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_op_history_size": "20", 2024-04-24T01:55:27.731 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_op_history_slow_op_size": "20", 2024-04-24T01:55:27.731 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_op_history_slow_op_threshold": "10", 2024-04-24T01:55:27.731 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_op_log_threshold": "5", 2024-04-24T01:55:27.731 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_ping_grace": "15", 2024-04-24T01:55:27.731 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_ping_interval": "5", 2024-04-24T01:55:27.731 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-04-24T01:55:27.731 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-04-24T01:55:27.732 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_recall_max_caps": "30000", 2024-04-24T01:55:27.732 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-04-24T01:55:27.732 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_recall_max_decay_threshold": "131072", 2024-04-24T01:55:27.732 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-04-24T01:55:27.732 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_recall_warning_threshold": "262144", 2024-04-24T01:55:27.732 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_reconnect_timeout": "45.000000", 2024-04-24T01:55:27.732 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_replay_interval": "1.000000", 2024-04-24T01:55:27.732 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-04-24T01:55:27.732 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-04-24T01:55:27.732 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_root_ino_gid": "0", 2024-04-24T01:55:27.732 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_root_ino_uid": "0", 2024-04-24T01:55:27.732 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-04-24T01:55:27.732 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_server_dispatch_client_request_delay": "0", 2024-04-24T01:55:27.732 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_server_dispatch_killpoint_random": "0.000000", 2024-04-24T01:55:27.732 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_session_blocklist_on_evict": "true", 2024-04-24T01:55:27.733 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_session_blocklist_on_timeout": "true", 2024-04-24T01:55:27.733 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-04-24T01:55:27.733 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-04-24T01:55:27.733 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-04-24T01:55:27.733 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-04-24T01:55:27.733 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-04-24T01:55:27.733 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_session_metadata_threshold": "16777216", 2024-04-24T01:55:27.733 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-04-24T01:55:27.733 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_shutdown_check": "0", 2024-04-24T01:55:27.733 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_skip_ino": "0", 2024-04-24T01:55:27.733 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_sleep_rank_change": "0.000000", 2024-04-24T01:55:27.733 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_snap_max_uid": "4294967294", 2024-04-24T01:55:27.733 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_snap_min_uid": "0", 2024-04-24T01:55:27.733 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_snap_rstat": "false", 2024-04-24T01:55:27.734 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_standby_replay_damaged": "false", 2024-04-24T01:55:27.734 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_symlink_recovery": "true", 2024-04-24T01:55:27.734 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_task_status_update_interval": "2.000000", 2024-04-24T01:55:27.734 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_thrash_exports": "0", 2024-04-24T01:55:27.734 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_thrash_fragments": "0", 2024-04-24T01:55:27.734 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_tick_interval": "5.000000", 2024-04-24T01:55:27.734 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_valgrind_exit": "false", 2024-04-24T01:55:27.734 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_verify_backtrace": "1", 2024-04-24T01:55:27.734 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_verify_scatter": "false", 2024-04-24T01:55:27.734 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_wipe_ino_prealloc": "false", 2024-04-24T01:55:27.734 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_wipe_sessions": "false", 2024-04-24T01:55:27.734 INFO:teuthology.orchestra.run.smithi079.stdout: "mempool_debug": "false", 2024-04-24T01:55:27.734 INFO:teuthology.orchestra.run.smithi079.stdout: "memstore_debug_omit_block_device_write": "false", 2024-04-24T01:55:27.734 INFO:teuthology.orchestra.run.smithi079.stdout: "memstore_device_bytes": "1073741824", 2024-04-24T01:55:27.735 INFO:teuthology.orchestra.run.smithi079.stdout: "memstore_page_set": "false", 2024-04-24T01:55:27.735 INFO:teuthology.orchestra.run.smithi079.stdout: "memstore_page_size": "65536", 2024-04-24T01:55:27.735 INFO:teuthology.orchestra.run.smithi079.stdout: "mgr_client_bytes": "134217728", 2024-04-24T01:55:27.735 INFO:teuthology.orchestra.run.smithi079.stdout: "mgr_client_messages": "512", 2024-04-24T01:55:27.735 INFO:teuthology.orchestra.run.smithi079.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-04-24T01:55:27.735 INFO:teuthology.orchestra.run.smithi079.stdout: "mgr_connect_retry_interval": "1.000000", 2024-04-24T01:55:27.735 INFO:teuthology.orchestra.run.smithi079.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-04-24T01:55:27.735 INFO:teuthology.orchestra.run.smithi079.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-04-24T01:55:27.735 INFO:teuthology.orchestra.run.smithi079.stdout: "mgr_disabled_modules": "", 2024-04-24T01:55:27.735 INFO:teuthology.orchestra.run.smithi079.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-04-24T01:55:27.735 INFO:teuthology.orchestra.run.smithi079.stdout: "mgr_max_pg_creating": "1024", 2024-04-24T01:55:27.735 INFO:teuthology.orchestra.run.smithi079.stdout: "mgr_max_pg_num_change": "128", 2024-04-24T01:55:27.735 INFO:teuthology.orchestra.run.smithi079.stdout: "mgr_mds_bytes": "134217728", 2024-04-24T01:55:27.735 INFO:teuthology.orchestra.run.smithi079.stdout: "mgr_mds_messages": "128", 2024-04-24T01:55:27.736 INFO:teuthology.orchestra.run.smithi079.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-04-24T01:55:27.736 INFO:teuthology.orchestra.run.smithi079.stdout: "mgr_mon_bytes": "134217728", 2024-04-24T01:55:27.736 INFO:teuthology.orchestra.run.smithi079.stdout: "mgr_mon_messages": "128", 2024-04-24T01:55:27.736 INFO:teuthology.orchestra.run.smithi079.stdout: "mgr_osd_bytes": "536870912", 2024-04-24T01:55:27.736 INFO:teuthology.orchestra.run.smithi079.stdout: "mgr_osd_messages": "8192", 2024-04-24T01:55:27.736 INFO:teuthology.orchestra.run.smithi079.stdout: "mgr_pool": "true", 2024-04-24T01:55:27.736 INFO:teuthology.orchestra.run.smithi079.stdout: "mgr_service_beacon_grace": "60.000000", 2024-04-24T01:55:27.736 INFO:teuthology.orchestra.run.smithi079.stdout: "mgr_standby_modules": "true", 2024-04-24T01:55:27.736 INFO:teuthology.orchestra.run.smithi079.stdout: "mgr_stats_period": "5", 2024-04-24T01:55:27.736 INFO:teuthology.orchestra.run.smithi079.stdout: "mgr_stats_threshold": "5", 2024-04-24T01:55:27.736 INFO:teuthology.orchestra.run.smithi079.stdout: "mgr_tick_period": "2", 2024-04-24T01:55:27.736 INFO:teuthology.orchestra.run.smithi079.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-04-24T01:55:27.736 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_accept_timeout_factor": "2.000000", 2024-04-24T01:55:27.736 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_allow_pool_delete": "true", 2024-04-24T01:55:27.737 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_allow_pool_size_one": "true", 2024-04-24T01:55:27.737 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_auth_validate_all_caps": "true", 2024-04-24T01:55:27.737 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-04-24T01:55:27.737 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-04-24T01:55:27.737 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_client_bytes": "104857600", 2024-04-24T01:55:27.737 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_client_directed_command_retry": "2", 2024-04-24T01:55:27.737 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_client_hunt_interval": "3.000000", 2024-04-24T01:55:27.737 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-04-24T01:55:27.737 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-04-24T01:55:27.737 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-04-24T01:55:27.737 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_client_hunt_parallel": "3", 2024-04-24T01:55:27.737 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_client_log_interval": "1.000000", 2024-04-24T01:55:27.737 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-04-24T01:55:27.737 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_client_ping_interval": "10.000000", 2024-04-24T01:55:27.737 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_client_ping_timeout": "30.000000", 2024-04-24T01:55:27.738 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_client_target_rank": "-1", 2024-04-24T01:55:27.738 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_clock_drift_allowed": "1.000000", 2024-04-24T01:55:27.738 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-04-24T01:55:27.738 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-04-24T01:55:27.738 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_cluster_log_level": "debug", 2024-04-24T01:55:27.738 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_cluster_log_to_file": "true", 2024-04-24T01:55:27.738 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_cluster_log_to_graylog": "false", 2024-04-24T01:55:27.738 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-04-24T01:55:27.738 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-04-24T01:55:27.738 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_cluster_log_to_journald": "false", 2024-04-24T01:55:27.738 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_cluster_log_to_stderr": "false", 2024-04-24T01:55:27.738 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-04-24T01:55:27.738 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-04-24T01:55:27.738 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_compact_on_bootstrap": "false", 2024-04-24T01:55:27.739 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_compact_on_start": "false", 2024-04-24T01:55:27.739 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_compact_on_trim": "true", 2024-04-24T01:55:27.739 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_con_tracker_persist_interval": "10", 2024-04-24T01:55:27.739 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_con_tracker_score_halflife": "43200", 2024-04-24T01:55:27.739 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_config_key_max_entry_size": "65536", 2024-04-24T01:55:27.739 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_cpu_threads": "4", 2024-04-24T01:55:27.739 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_crush_min_required_version": "hammer", 2024-04-24T01:55:27.739 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_daemon_bytes": "419430400", 2024-04-24T01:55:27.739 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-04-24T01:55:27.739 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_data_avail_crit": "5", 2024-04-24T01:55:27.739 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_data_avail_warn": "5", 2024-04-24T01:55:27.739 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_data_size_warn": "16106127360", 2024-04-24T01:55:27.739 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_debug_block_osdmap_trim": "false", 2024-04-24T01:55:27.739 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-04-24T01:55:27.740 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_debug_dump_json": "false", 2024-04-24T01:55:27.740 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-04-24T01:55:27.740 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_debug_dump_transactions": "false", 2024-04-24T01:55:27.740 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_debug_extra_checks": "false", 2024-04-24T01:55:27.740 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-04-24T01:55:27.740 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-04-24T01:55:27.740 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_debug_no_require_reef": "false", 2024-04-24T01:55:27.740 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_debug_no_require_squid": "false", 2024-04-24T01:55:27.740 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-04-24T01:55:27.740 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_delta_reset_interval": "10.000000", 2024-04-24T01:55:27.740 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_dns_srv_name": "ceph-mon", 2024-04-24T01:55:27.740 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_down_mkfs_grace": "120", 2024-04-24T01:55:27.740 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_down_uptime_grace": "60", 2024-04-24T01:55:27.740 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_election_default_strategy": "1", 2024-04-24T01:55:27.741 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_election_timeout": "5.000000", 2024-04-24T01:55:27.741 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-04-24T01:55:27.741 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_elector_ping_divisor": "2", 2024-04-24T01:55:27.741 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_elector_ping_timeout": "2.000000", 2024-04-24T01:55:27.741 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_enable_op_tracker": "true", 2024-04-24T01:55:27.741 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_fake_pool_delete": "false", 2024-04-24T01:55:27.741 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_force_quorum_join": "false", 2024-04-24T01:55:27.741 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_globalid_prealloc": "10000", 2024-04-24T01:55:27.741 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_health_detail_to_clog": "false", 2024-04-24T01:55:27.741 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_health_log_update_period": "5", 2024-04-24T01:55:27.741 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_health_max_detail": "50", 2024-04-24T01:55:27.741 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_health_to_clog": "true", 2024-04-24T01:55:27.741 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_health_to_clog_interval": "600", 2024-04-24T01:55:27.741 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-04-24T01:55:27.741 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_host": "172.21.15.79", 2024-04-24T01:55:27.742 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_host_override": "", 2024-04-24T01:55:27.742 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_initial_members": "", 2024-04-24T01:55:27.742 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-04-24T01:55:27.742 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-04-24T01:55:27.742 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-04-24T01:55:27.742 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-04-24T01:55:27.742 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_keyvaluedb": "rocksdb", 2024-04-24T01:55:27.742 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_lease": "5.000000", 2024-04-24T01:55:27.742 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-04-24T01:55:27.742 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-04-24T01:55:27.742 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_log_full_interval": "50", 2024-04-24T01:55:27.742 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_log_max": "10000", 2024-04-24T01:55:27.742 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_log_max_summary": "50", 2024-04-24T01:55:27.742 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_max_log_entries_per_event": "4096", 2024-04-24T01:55:27.743 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_max_log_epochs": "500", 2024-04-24T01:55:27.743 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_max_mdsmap_epochs": "500", 2024-04-24T01:55:27.743 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_max_mgrmap_epochs": "500", 2024-04-24T01:55:27.743 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_max_osd": "10000", 2024-04-24T01:55:27.743 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_max_pg_per_osd": "10000", 2024-04-24T01:55:27.743 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_max_pool_pg_num": "65536", 2024-04-24T01:55:27.743 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-04-24T01:55:27.743 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-04-24T01:55:27.743 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_mds_force_trim_to": "0", 2024-04-24T01:55:27.743 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_mds_skip_sanity": "false", 2024-04-24T01:55:27.743 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_memory_autotune": "true", 2024-04-24T01:55:27.743 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_memory_target": "2147483648", 2024-04-24T01:55:27.743 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_mgr_beacon_grace": "30", 2024-04-24T01:55:27.744 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-04-24T01:55:27.744 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_mgr_digest_period": "5", 2024-04-24T01:55:27.744 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_mgr_inactive_grace": "60", 2024-04-24T01:55:27.744 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_mgr_mkfs_grace": "240", 2024-04-24T01:55:27.744 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-04-24T01:55:27.744 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_min_osdmap_epochs": "50", 2024-04-24T01:55:27.744 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_op_complaint_time": "30", 2024-04-24T01:55:27.744 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_op_history_duration": "600", 2024-04-24T01:55:27.744 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_op_history_size": "20", 2024-04-24T01:55:27.744 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_op_history_slow_op_size": "20", 2024-04-24T01:55:27.744 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_op_history_slow_op_threshold": "10", 2024-04-24T01:55:27.744 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_op_log_threshold": "5", 2024-04-24T01:55:27.744 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-04-24T01:55:27.744 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-04-24T01:55:27.744 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-04-24T01:55:27.745 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_osd_auto_mark_in": "false", 2024-04-24T01:55:27.745 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_osd_auto_mark_new_in": "true", 2024-04-24T01:55:27.745 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-04-24T01:55:27.745 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-04-24T01:55:27.745 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_osd_cache_size": "500", 2024-04-24T01:55:27.745 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_osd_cache_size_min": "134217728", 2024-04-24T01:55:27.745 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_osd_crush_smoke_test": "true", 2024-04-24T01:55:27.745 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_osd_destroyed_out_interval": "600", 2024-04-24T01:55:27.745 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_osd_down_out_interval": "600", 2024-04-24T01:55:27.745 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-04-24T01:55:27.745 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-04-24T01:55:27.745 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_osd_force_trim_to": "0", 2024-04-24T01:55:27.745 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_osd_full_ratio": "0.950000", 2024-04-24T01:55:27.745 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-04-24T01:55:27.746 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_osd_laggy_halflife": "3600", 2024-04-24T01:55:27.746 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_osd_laggy_max_interval": "300", 2024-04-24T01:55:27.746 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_osd_laggy_weight": "0.300000", 2024-04-24T01:55:27.746 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-04-24T01:55:27.746 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_osd_max_initial_pgs": "1024", 2024-04-24T01:55:27.746 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_osd_min_down_reporters": "2", 2024-04-24T01:55:27.746 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-04-24T01:55:27.746 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-04-24T01:55:27.746 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-04-24T01:55:27.746 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_osd_prime_pg_temp": "true", 2024-04-24T01:55:27.746 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-04-24T01:55:27.746 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-04-24T01:55:27.746 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_osd_report_timeout": "900", 2024-04-24T01:55:27.747 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-04-24T01:55:27.747 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-04-24T01:55:27.747 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_osd_warn_num_repaired": "10", 2024-04-24T01:55:27.747 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_osd_warn_op_age": "32.000000", 2024-04-24T01:55:27.747 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-04-24T01:55:27.747 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_osdmap_full_prune_interval": "2", 2024-04-24T01:55:27.747 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_osdmap_full_prune_min": "15", 2024-04-24T01:55:27.747 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_osdmap_full_prune_txsize": "2", 2024-04-24T01:55:27.747 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-04-24T01:55:27.747 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_pg_stuck_threshold": "60", 2024-04-24T01:55:27.747 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-04-24T01:55:27.747 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_pg_warn_min_objects": "10000", 2024-04-24T01:55:27.747 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_pg_warn_min_per_osd": "0", 2024-04-24T01:55:27.747 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-04-24T01:55:27.747 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_pool_quota_crit_threshold": "0", 2024-04-24T01:55:27.748 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_pool_quota_warn_threshold": "0", 2024-04-24T01:55:27.748 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_probe_timeout": "2.000000", 2024-04-24T01:55:27.748 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_reweight_max_change": "0.050000", 2024-04-24T01:55:27.748 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_reweight_max_osds": "4", 2024-04-24T01:55:27.748 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-04-24T01:55:27.748 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-04-24T01:55:27.748 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-04-24T01:55:27.748 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-04-24T01:55:27.748 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-04-24T01:55:27.748 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_scrub_interval": "86400", 2024-04-24T01:55:27.748 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_scrub_max_keys": "100", 2024-04-24T01:55:27.748 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_scrub_timeout": "300", 2024-04-24T01:55:27.748 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_session_timeout": "300", 2024-04-24T01:55:27.748 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_smart_report_timeout": "5", 2024-04-24T01:55:27.749 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_stat_smooth_intervals": "6", 2024-04-24T01:55:27.749 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-04-24T01:55:27.749 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_stretch_pool_min_size": "2", 2024-04-24T01:55:27.749 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_stretch_pool_size": "4", 2024-04-24T01:55:27.749 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-04-24T01:55:27.749 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_subscribe_interval": "86400.000000", 2024-04-24T01:55:27.749 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_sync_debug": "false", 2024-04-24T01:55:27.749 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_sync_max_payload_keys": "2000", 2024-04-24T01:55:27.749 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_sync_max_payload_size": "1048576", 2024-04-24T01:55:27.749 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_sync_provider_kill_at": "0", 2024-04-24T01:55:27.749 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_sync_requester_kill_at": "0", 2024-04-24T01:55:27.749 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_sync_timeout": "60.000000", 2024-04-24T01:55:27.749 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_target_pg_per_osd": "100", 2024-04-24T01:55:27.749 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_tick_interval": "5", 2024-04-24T01:55:27.750 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_timecheck_interval": "300.000000", 2024-04-24T01:55:27.750 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-04-24T01:55:27.750 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_use_min_delay_socket": "false", 2024-04-24T01:55:27.750 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_warn_older_version_delay": "604800", 2024-04-24T01:55:27.750 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-04-24T01:55:27.750 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-04-24T01:55:27.750 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-04-24T01:55:27.750 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_warn_on_filestore_osds": "false", 2024-04-24T01:55:27.750 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-04-24T01:55:27.750 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-04-24T01:55:27.750 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-04-24T01:55:27.750 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_warn_on_misplaced": "false", 2024-04-24T01:55:27.750 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-04-24T01:55:27.750 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_warn_on_older_version": "true", 2024-04-24T01:55:27.751 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-04-24T01:55:27.751 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_warn_on_pool_no_app": "true", 2024-04-24T01:55:27.751 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_warn_on_pool_no_redundancy": "false", 2024-04-24T01:55:27.751 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-04-24T01:55:27.751 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-04-24T01:55:27.751 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-04-24T01:55:27.751 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_warn_on_too_few_osds": "false", 2024-04-24T01:55:27.751 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-04-24T01:55:27.751 INFO:teuthology.orchestra.run.smithi079.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-04-24T01:55:27.751 INFO:teuthology.orchestra.run.smithi079.stdout: "monmap": "", 2024-04-24T01:55:27.751 INFO:teuthology.orchestra.run.smithi079.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-04-24T01:55:27.751 INFO:teuthology.orchestra.run.smithi079.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-04-24T01:55:27.751 INFO:teuthology.orchestra.run.smithi079.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-04-24T01:55:27.751 INFO:teuthology.orchestra.run.smithi079.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-04-24T01:55:27.752 INFO:teuthology.orchestra.run.smithi079.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-04-24T01:55:27.752 INFO:teuthology.orchestra.run.smithi079.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-04-24T01:55:27.752 INFO:teuthology.orchestra.run.smithi079.stdout: "motr_tracing_enabled": "false", 2024-04-24T01:55:27.752 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_async_op_threads": "3", 2024-04-24T01:55:27.752 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_async_rdma_buffer_size": "131072", 2024-04-24T01:55:27.752 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_async_rdma_cm": "false", 2024-04-24T01:55:27.752 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_async_rdma_device_name": "", 2024-04-24T01:55:27.752 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_async_rdma_dscp": "96", 2024-04-24T01:55:27.752 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-04-24T01:55:27.752 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_async_rdma_gid_idx": "0", 2024-04-24T01:55:27.752 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_async_rdma_local_gid": "", 2024-04-24T01:55:27.752 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_async_rdma_polling_us": "1000", 2024-04-24T01:55:27.752 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_async_rdma_port_num": "1", 2024-04-24T01:55:27.752 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-04-24T01:55:27.753 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-04-24T01:55:27.753 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_async_rdma_roce_ver": "1", 2024-04-24T01:55:27.753 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_async_rdma_send_buffers": "1024", 2024-04-24T01:55:27.753 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_async_rdma_sl": "3", 2024-04-24T01:55:27.753 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_async_rdma_support_srq": "true", 2024-04-24T01:55:27.753 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_async_rdma_type": "ib", 2024-04-24T01:55:27.753 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_async_reap_threshold": "5", 2024-04-24T01:55:27.753 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_bind_before_connect": "false", 2024-04-24T01:55:27.753 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_bind_ipv4": "true", 2024-04-24T01:55:27.753 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_bind_ipv6": "false", 2024-04-24T01:55:27.753 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_bind_msgr1": "true", 2024-04-24T01:55:27.753 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_bind_msgr2": "true", 2024-04-24T01:55:27.753 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_bind_port_max": "7568", 2024-04-24T01:55:27.753 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_bind_port_min": "6800", 2024-04-24T01:55:27.754 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_bind_prefer_ipv4": "false", 2024-04-24T01:55:27.754 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_bind_retry_count": "3", 2024-04-24T01:55:27.754 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_bind_retry_delay": "5", 2024-04-24T01:55:27.754 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_blackhole_client": "false", 2024-04-24T01:55:27.754 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_blackhole_mds": "false", 2024-04-24T01:55:27.754 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_blackhole_mgr": "false", 2024-04-24T01:55:27.754 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_blackhole_mon": "false", 2024-04-24T01:55:27.754 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_blackhole_osd": "false", 2024-04-24T01:55:27.754 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_client_mode": "crc", 2024-04-24T01:55:27.754 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_cluster_mode": "crc", 2024-04-24T01:55:27.754 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_cluster_type": "", 2024-04-24T01:55:27.754 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_compress_secure": "false", 2024-04-24T01:55:27.754 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_connection_idle_timeout": "900", 2024-04-24T01:55:27.754 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_connection_ready_timeout": "10", 2024-04-24T01:55:27.755 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_crc_data": "true", 2024-04-24T01:55:27.755 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_crc_header": "true", 2024-04-24T01:55:27.755 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_die_on_bad_msg": "false", 2024-04-24T01:55:27.755 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_die_on_bug": "true", 2024-04-24T01:55:27.755 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_die_on_old_message": "true", 2024-04-24T01:55:27.755 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_die_on_skipped_message": "false", 2024-04-24T01:55:27.755 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_die_on_unhandled_msg": "false", 2024-04-24T01:55:27.755 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-04-24T01:55:27.755 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_dpdk_coremask": "0xF", 2024-04-24T01:55:27.755 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-04-24T01:55:27.755 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_dpdk_devs_allowlist": "", 2024-04-24T01:55:27.755 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_dpdk_enable_tso": "true", 2024-04-24T01:55:27.755 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-04-24T01:55:27.755 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-04-24T01:55:27.755 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_dpdk_hugepages": "", 2024-04-24T01:55:27.756 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_dpdk_hw_flow_control": "true", 2024-04-24T01:55:27.756 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-04-24T01:55:27.756 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_dpdk_lro": "true", 2024-04-24T01:55:27.756 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_dpdk_memory_channel": "4", 2024-04-24T01:55:27.756 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-04-24T01:55:27.756 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_dpdk_pmd": "", 2024-04-24T01:55:27.756 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_dpdk_port_id": "0", 2024-04-24T01:55:27.756 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-04-24T01:55:27.756 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_dump_corrupt_message_level": "1", 2024-04-24T01:55:27.756 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_dump_on_send": "false", 2024-04-24T01:55:27.756 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_initial_backoff": "0.200000", 2024-04-24T01:55:27.756 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_inject_delay_max": "1.000000", 2024-04-24T01:55:27.756 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_inject_delay_probability": "0.000000", 2024-04-24T01:55:27.756 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_inject_delay_type": "", 2024-04-24T01:55:27.757 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_inject_internal_delays": "0.000000", 2024-04-24T01:55:27.757 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_inject_network_congestion": "0", 2024-04-24T01:55:27.757 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_inject_socket_failures": "0", 2024-04-24T01:55:27.757 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_learn_addr_from_peer": "true", 2024-04-24T01:55:27.757 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_max_accept_failures": "4", 2024-04-24T01:55:27.757 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_max_backoff": "15.000000", 2024-04-24T01:55:27.757 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_mon_client_mode": "crc", 2024-04-24T01:55:27.757 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_mon_cluster_mode": "crc", 2024-04-24T01:55:27.757 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_mon_service_mode": "crc", 2024-04-24T01:55:27.757 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_osd_compress_min_size": "1024", 2024-04-24T01:55:27.757 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_osd_compress_mode": "none", 2024-04-24T01:55:27.757 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_osd_compression_algorithm": "snappy", 2024-04-24T01:55:27.757 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-04-24T01:55:27.757 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_pq_min_cost": "65536", 2024-04-24T01:55:27.757 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_public_type": "", 2024-04-24T01:55:27.758 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_service_mode": "crc", 2024-04-24T01:55:27.758 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_tcp_listen_backlog": "512", 2024-04-24T01:55:27.758 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_tcp_nodelay": "true", 2024-04-24T01:55:27.758 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-04-24T01:55:27.758 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_tcp_rcvbuf": "0", 2024-04-24T01:55:27.758 INFO:teuthology.orchestra.run.smithi079.stdout: "ms_type": "async+posix", 2024-04-24T01:55:27.758 INFO:teuthology.orchestra.run.smithi079.stdout: "no_config_file": "false", 2024-04-24T01:55:27.758 INFO:teuthology.orchestra.run.smithi079.stdout: "objecter_completion_locks_per_session": "32", 2024-04-24T01:55:27.758 INFO:teuthology.orchestra.run.smithi079.stdout: "objecter_debug_inject_relock_delay": "false", 2024-04-24T01:55:27.758 INFO:teuthology.orchestra.run.smithi079.stdout: "objecter_inflight_op_bytes": "104857600", 2024-04-24T01:55:27.758 INFO:teuthology.orchestra.run.smithi079.stdout: "objecter_inflight_ops": "1024", 2024-04-24T01:55:27.758 INFO:teuthology.orchestra.run.smithi079.stdout: "objecter_inject_no_watch_ping": "false", 2024-04-24T01:55:27.758 INFO:teuthology.orchestra.run.smithi079.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-04-24T01:55:27.758 INFO:teuthology.orchestra.run.smithi079.stdout: "objecter_tick_interval": "5.000000", 2024-04-24T01:55:27.759 INFO:teuthology.orchestra.run.smithi079.stdout: "objecter_timeout": "10.000000", 2024-04-24T01:55:27.759 INFO:teuthology.orchestra.run.smithi079.stdout: "objectstore_blackhole": "false", 2024-04-24T01:55:27.759 INFO:teuthology.orchestra.run.smithi079.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-04-24T01:55:27.759 INFO:teuthology.orchestra.run.smithi079.stdout: "openssl_engine_opts": "", 2024-04-24T01:55:27.759 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_agent_delay_time": "5.000000", 2024-04-24T01:55:27.759 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_agent_hist_halflife": "1000", 2024-04-24T01:55:27.759 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_agent_max_low_ops": "2", 2024-04-24T01:55:27.759 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_agent_max_ops": "4", 2024-04-24T01:55:27.759 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-04-24T01:55:27.759 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_agent_quantize_effort": "0.100000", 2024-04-24T01:55:27.759 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_agent_slop": "0.020000", 2024-04-24T01:55:27.759 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-04-24T01:55:27.759 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_allow_recovery_below_min_size": "true", 2024-04-24T01:55:27.759 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_asio_thread_count": "2", 2024-04-24T01:55:27.760 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_async_recovery_min_cost": "100", 2024-04-24T01:55:27.760 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_auto_mark_unfound_lost": "false", 2024-04-24T01:55:27.760 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_backfill_retry_interval": "30.000000", 2024-04-24T01:55:27.760 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_backfill_scan_max": "512", 2024-04-24T01:55:27.760 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_backfill_scan_min": "64", 2024-04-24T01:55:27.760 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_backoff_on_degraded": "false", 2024-04-24T01:55:27.760 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_backoff_on_peering": "false", 2024-04-24T01:55:27.760 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_backoff_on_unfound": "true", 2024-04-24T01:55:27.760 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_beacon_report_interval": "300", 2024-04-24T01:55:27.760 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_bench_duration": "30", 2024-04-24T01:55:27.760 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-04-24T01:55:27.760 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_bench_max_block_size": "67108864", 2024-04-24T01:55:27.760 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_bench_small_size_max_iops": "100", 2024-04-24T01:55:27.760 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_blkin_trace_all": "false", 2024-04-24T01:55:27.760 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_blocked_scrub_grace_period": "120", 2024-04-24T01:55:27.761 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-04-24T01:55:27.761 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-04-24T01:55:27.761 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-04-24T01:55:27.761 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_check_for_log_corruption": "false", 2024-04-24T01:55:27.761 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-04-24T01:55:27.761 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-04-24T01:55:27.761 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-04-24T01:55:27.761 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-04-24T01:55:27.761 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_class_update_on_start": "true", 2024-04-24T01:55:27.761 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_client_message_cap": "256", 2024-04-24T01:55:27.761 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_client_message_size_cap": "524288000", 2024-04-24T01:55:27.761 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_client_op_priority": "63", 2024-04-24T01:55:27.761 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_client_watch_timeout": "30", 2024-04-24T01:55:27.761 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_command_max_records": "256", 2024-04-24T01:55:27.762 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_compact_on_start": "false", 2024-04-24T01:55:27.762 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-04-24T01:55:27.762 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_crush_chooseleaf_type": "0", 2024-04-24T01:55:27.762 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_crush_initial_weight": "-1.000000", 2024-04-24T01:55:27.762 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_crush_update_on_start": "true", 2024-04-24T01:55:27.762 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_crush_update_weight_set": "true", 2024-04-24T01:55:27.762 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-04-24T01:55:27.762 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-04-24T01:55:27.762 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-04-24T01:55:27.762 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_debug_drop_ping_duration": "0", 2024-04-24T01:55:27.762 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-04-24T01:55:27.762 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_debug_feed_pullee": "-1", 2024-04-24T01:55:27.762 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-04-24T01:55:27.762 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-04-24T01:55:27.762 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-04-24T01:55:27.763 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_debug_misdirected_ops": "false", 2024-04-24T01:55:27.763 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_debug_no_acting_change": "false", 2024-04-24T01:55:27.763 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_debug_no_purge_strays": "false", 2024-04-24T01:55:27.763 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_debug_op_order": "false", 2024-04-24T01:55:27.763 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_debug_pg_log_writeout": "false", 2024-04-24T01:55:27.763 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_debug_pretend_recovery_active": "false", 2024-04-24T01:55:27.763 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-04-24T01:55:27.763 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-04-24T01:55:27.763 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_debug_shutdown": "false", 2024-04-24T01:55:27.763 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-04-24T01:55:27.763 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-04-24T01:55:27.763 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_debug_trim_objects": "false", 2024-04-24T01:55:27.763 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_debug_verify_cached_snaps": "false", 2024-04-24T01:55:27.763 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_debug_verify_missing_on_start": "false", 2024-04-24T01:55:27.764 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_debug_verify_snaps": "false", 2024-04-24T01:55:27.764 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-04-24T01:55:27.764 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-04-24T01:55:27.764 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_deep_scrub_keys": "1024", 2024-04-24T01:55:27.764 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-04-24T01:55:27.764 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-04-24T01:55:27.764 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-04-24T01:55:27.764 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_deep_scrub_stride": "524288", 2024-04-24T01:55:27.764 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-04-24T01:55:27.764 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_default_data_pool_replay_window": "5", 2024-04-24T01:55:27.764 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_default_notify_timeout": "30", 2024-04-24T01:55:27.764 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_delete_sleep": "0.000000", 2024-04-24T01:55:27.764 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-04-24T01:55:27.764 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-04-24T01:55:27.764 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-04-24T01:55:27.765 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_discard_disconnected_ops": "true", 2024-04-24T01:55:27.765 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_enable_op_tracker": "true", 2024-04-24T01:55:27.765 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-04-24T01:55:27.765 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_extblkdev_plugins": "vdo", 2024-04-24T01:55:27.765 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-04-24T01:55:27.765 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-04-24T01:55:27.765 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_fast_info": "true", 2024-04-24T01:55:27.765 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_fast_shutdown": "true", 2024-04-24T01:55:27.765 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-04-24T01:55:27.765 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_fast_shutdown_timeout": "15", 2024-04-24T01:55:27.765 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-04-24T01:55:27.765 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-04-24T01:55:27.765 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-04-24T01:55:27.765 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_function_tracing": "false", 2024-04-24T01:55:27.765 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_heartbeat_grace": "20", 2024-04-24T01:55:27.766 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_heartbeat_interval": "6", 2024-04-24T01:55:27.766 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-04-24T01:55:27.766 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_heartbeat_min_peers": "10", 2024-04-24T01:55:27.766 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_heartbeat_min_size": "2000", 2024-04-24T01:55:27.766 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_heartbeat_stale": "600", 2024-04-24T01:55:27.766 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-04-24T01:55:27.766 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_hit_set_max_size": "100000", 2024-04-24T01:55:27.766 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_hit_set_min_size": "1000", 2024-04-24T01:55:27.766 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-04-24T01:55:27.766 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-04-24T01:55:27.766 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-04-24T01:55:27.766 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-04-24T01:55:27.766 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-04-24T01:55:27.766 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_journal_flush_on_shutdown": "true", 2024-04-24T01:55:27.767 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_journal_size": "5120", 2024-04-24T01:55:27.767 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_kill_backfill_at": "0", 2024-04-24T01:55:27.767 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_loop_before_reset_tphandle": "64", 2024-04-24T01:55:27.767 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_map_cache_size": "50", 2024-04-24T01:55:27.767 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_map_dedup": "true", 2024-04-24T01:55:27.767 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_map_message_max": "40", 2024-04-24T01:55:27.767 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_map_message_max_bytes": "10485760", 2024-04-24T01:55:27.767 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_map_share_max_epochs": "40", 2024-04-24T01:55:27.767 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_max_attr_name_len": "100", 2024-04-24T01:55:27.767 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_max_attr_size": "0", 2024-04-24T01:55:27.767 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_max_backfills": "1", 2024-04-24T01:55:27.767 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_max_markdown_count": "5", 2024-04-24T01:55:27.767 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_max_markdown_period": "600", 2024-04-24T01:55:27.767 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_max_object_name_len": "2048", 2024-04-24T01:55:27.767 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_max_object_namespace_len": "256", 2024-04-24T01:55:27.768 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_max_object_size": "134217728", 2024-04-24T01:55:27.768 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-04-24T01:55:27.768 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_max_omap_entries_per_request": "1024", 2024-04-24T01:55:27.768 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_max_pg_blocked_by": "16", 2024-04-24T01:55:27.768 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_max_pg_log_entries": "10000", 2024-04-24T01:55:27.768 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-04-24T01:55:27.768 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_max_pgls": "1024", 2024-04-24T01:55:27.768 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_max_push_cost": "8388608", 2024-04-24T01:55:27.768 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_max_push_objects": "10", 2024-04-24T01:55:27.768 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_max_scrubs": "3", 2024-04-24T01:55:27.768 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-04-24T01:55:27.768 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_max_trimming_pgs": "2", 2024-04-24T01:55:27.768 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_max_write_op_reply_len": "64", 2024-04-24T01:55:27.768 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_max_write_size": "90", 2024-04-24T01:55:27.768 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-04-24T01:55:27.769 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-04-24T01:55:27.769 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-04-24T01:55:27.769 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-04-24T01:55:27.769 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-04-24T01:55:27.769 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-04-24T01:55:27.769 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-04-24T01:55:27.769 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_mclock_override_recovery_settings": "false", 2024-04-24T01:55:27.769 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_mclock_profile": "balanced", 2024-04-24T01:55:27.769 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-04-24T01:55:27.769 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-04-24T01:55:27.769 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-04-24T01:55:27.769 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-04-24T01:55:27.769 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-04-24T01:55:27.769 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-04-24T01:55:27.770 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-04-24T01:55:27.770 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-04-24T01:55:27.770 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-04-24T01:55:27.770 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-04-24T01:55:27.770 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_mclock_skip_benchmark": "false", 2024-04-24T01:55:27.770 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_memory_base": "805306368", 2024-04-24T01:55:27.770 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_memory_cache_min": "134217728", 2024-04-24T01:55:27.770 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-04-24T01:55:27.770 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-04-24T01:55:27.770 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_memory_target": "4294967296", 2024-04-24T01:55:27.770 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_memory_target_autotune": "false", 2024-04-24T01:55:27.770 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-04-24T01:55:27.770 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_min_pg_log_entries": "250", 2024-04-24T01:55:27.770 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_min_recovery_priority": "0", 2024-04-24T01:55:27.771 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_mon_heartbeat_interval": "30", 2024-04-24T01:55:27.771 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-04-24T01:55:27.771 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_mon_report_interval": "5", 2024-04-24T01:55:27.771 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_mon_report_max_in_flight": "2", 2024-04-24T01:55:27.771 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-04-24T01:55:27.771 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_num_cache_shards": "32", 2024-04-24T01:55:27.771 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_num_op_tracker_shard": "32", 2024-04-24T01:55:27.771 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_numa_auto_affinity": "true", 2024-04-24T01:55:27.771 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_numa_node": "-1", 2024-04-24T01:55:27.771 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_numa_prefer_iface": "true", 2024-04-24T01:55:27.771 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-04-24T01:55:27.771 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_objecter_finishers": "1", 2024-04-24T01:55:27.771 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_objectstore": "bluestore", 2024-04-24T01:55:27.771 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_objectstore_fuse": "false", 2024-04-24T01:55:27.771 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_objectstore_tracing": "false", 2024-04-24T01:55:27.772 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_op_complaint_time": "30.000000", 2024-04-24T01:55:27.772 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_op_history_duration": "600", 2024-04-24T01:55:27.772 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_op_history_size": "20", 2024-04-24T01:55:27.772 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_op_history_slow_op_size": "20", 2024-04-24T01:55:27.772 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-04-24T01:55:27.772 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_op_log_threshold": "5", 2024-04-24T01:55:27.772 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_op_num_shards": "0", 2024-04-24T01:55:27.772 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_op_num_shards_hdd": "5", 2024-04-24T01:55:27.772 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_op_num_shards_ssd": "8", 2024-04-24T01:55:27.772 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_op_num_threads_per_shard": "0", 2024-04-24T01:55:27.772 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-04-24T01:55:27.772 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-04-24T01:55:27.772 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-04-24T01:55:27.772 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_op_pq_min_cost": "65536", 2024-04-24T01:55:27.772 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_op_queue": "mclock_scheduler", 2024-04-24T01:55:27.773 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_op_queue_cut_off": "high", 2024-04-24T01:55:27.773 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_op_thread_suicide_timeout": "150", 2024-04-24T01:55:27.773 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_op_thread_timeout": "15", 2024-04-24T01:55:27.773 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_open_classes_on_start": "true", 2024-04-24T01:55:27.773 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_os_flags": "0", 2024-04-24T01:55:27.773 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_peering_op_priority": "255", 2024-04-24T01:55:27.773 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_pg_delete_cost": "1048576", 2024-04-24T01:55:27.773 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_pg_delete_priority": "5", 2024-04-24T01:55:27.773 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-04-24T01:55:27.773 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-04-24T01:55:27.773 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_pg_log_dups_tracked": "3000", 2024-04-24T01:55:27.773 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_pg_log_trim_max": "10000", 2024-04-24T01:55:27.773 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_pg_log_trim_min": "100", 2024-04-24T01:55:27.773 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-04-24T01:55:27.773 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_pg_object_context_cache_count": "64", 2024-04-24T01:55:27.774 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-04-24T01:55:27.774 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2024-04-24T01:55:27.774 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-04-24T01:55:27.774 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-04-24T01:55:27.774 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-04-24T01:55:27.774 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-04-24T01:55:27.774 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-04-24T01:55:27.774 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-04-24T01:55:27.774 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_pool_default_crimson": "true", 2024-04-24T01:55:27.774 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_pool_default_crush_rule": "-1", 2024-04-24T01:55:27.774 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_pool_default_ec_fast_read": "false", 2024-04-24T01:55:27.774 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-04-24T01:55:27.774 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_pool_default_flag_bulk": "false", 2024-04-24T01:55:27.774 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-04-24T01:55:27.775 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_pool_default_flag_nodelete": "false", 2024-04-24T01:55:27.775 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-04-24T01:55:27.775 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-04-24T01:55:27.775 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_pool_default_flags": "0", 2024-04-24T01:55:27.775 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-04-24T01:55:27.775 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_pool_default_min_size": "2", 2024-04-24T01:55:27.775 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-04-24T01:55:27.775 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_pool_default_pg_num": "32", 2024-04-24T01:55:27.775 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_pool_default_pgp_num": "0", 2024-04-24T01:55:27.775 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-04-24T01:55:27.775 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_pool_default_read_ratio": "70", 2024-04-24T01:55:27.775 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_pool_default_size": "3", 2024-04-24T01:55:27.775 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_pool_default_type": "replicated", 2024-04-24T01:55:27.776 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-04-24T01:55:27.776 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_pool_use_gmt_hitset": "true", 2024-04-24T01:55:27.776 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_push_per_object_cost": "1000", 2024-04-24T01:55:27.776 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_read_ec_check_for_errors": "false", 2024-04-24T01:55:27.776 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_recover_clone_overlap": "true", 2024-04-24T01:55:27.776 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_recover_clone_overlap_limit": "10", 2024-04-24T01:55:27.776 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_recovery_cost": "20971520", 2024-04-24T01:55:27.776 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_recovery_delay_start": "0.000000", 2024-04-24T01:55:27.776 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_recovery_max_active": "0", 2024-04-24T01:55:27.776 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_recovery_max_active_hdd": "3", 2024-04-24T01:55:27.776 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_recovery_max_active_ssd": "10", 2024-04-24T01:55:27.776 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_recovery_max_chunk": "8388608", 2024-04-24T01:55:27.776 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-04-24T01:55:27.776 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_recovery_max_single_start": "1", 2024-04-24T01:55:27.776 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_recovery_op_priority": "3", 2024-04-24T01:55:27.777 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_recovery_op_warn_multiple": "16", 2024-04-24T01:55:27.777 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_recovery_priority": "5", 2024-04-24T01:55:27.777 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_recovery_retry_interval": "30.000000", 2024-04-24T01:55:27.777 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_recovery_sleep": "0.000000", 2024-04-24T01:55:27.777 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-04-24T01:55:27.777 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-04-24T01:55:27.777 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-04-24T01:55:27.777 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_repair_during_recovery": "false", 2024-04-24T01:55:27.777 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_requested_scrub_priority": "120", 2024-04-24T01:55:27.777 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-04-24T01:55:27.777 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_rollback_to_cluster_snap": "", 2024-04-24T01:55:27.777 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_scrub_auto_repair": "false", 2024-04-24T01:55:27.777 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-04-24T01:55:27.777 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-04-24T01:55:27.778 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_scrub_begin_hour": "0", 2024-04-24T01:55:27.778 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_scrub_begin_week_day": "0", 2024-04-24T01:55:27.778 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_scrub_chunk_max": "25", 2024-04-24T01:55:27.778 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_scrub_chunk_min": "5", 2024-04-24T01:55:27.778 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_scrub_cost": "52428800", 2024-04-24T01:55:27.778 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_scrub_disable_reservation_queuing": "false", 2024-04-24T01:55:27.778 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_scrub_during_recovery": "false", 2024-04-24T01:55:27.778 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_scrub_end_hour": "0", 2024-04-24T01:55:27.778 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_scrub_end_week_day": "0", 2024-04-24T01:55:27.778 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_scrub_event_cost": "4096", 2024-04-24T01:55:27.778 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-04-24T01:55:27.778 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-04-24T01:55:27.778 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_scrub_invalid_stats": "true", 2024-04-24T01:55:27.778 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_scrub_load_threshold": "0.500000", 2024-04-24T01:55:27.779 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_scrub_max_interval": "604800.000000", 2024-04-24T01:55:27.779 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_scrub_max_preemptions": "5", 2024-04-24T01:55:27.779 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_scrub_min_interval": "86400.000000", 2024-04-24T01:55:27.779 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_scrub_priority": "5", 2024-04-24T01:55:27.779 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_scrub_reservation_timeout": "300000", 2024-04-24T01:55:27.779 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_scrub_sleep": "0.000000", 2024-04-24T01:55:27.779 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-04-24T01:55:27.779 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-04-24T01:55:27.779 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_shutdown_pgref_assert": "false", 2024-04-24T01:55:27.779 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_skip_check_past_interval_bounds": "false", 2024-04-24T01:55:27.779 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_skip_data_digest": "false", 2024-04-24T01:55:27.779 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_smart_report_timeout": "5", 2024-04-24T01:55:27.779 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_snap_trim_cost": "1048576", 2024-04-24T01:55:27.779 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_snap_trim_priority": "5", 2024-04-24T01:55:27.779 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_snap_trim_sleep": "0.000000", 2024-04-24T01:55:27.780 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-04-24T01:55:27.780 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-04-24T01:55:27.780 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-04-24T01:55:27.780 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-04-24T01:55:27.780 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_stats_update_period_scrubbing": "15", 2024-04-24T01:55:27.780 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-04-24T01:55:27.780 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_target_transaction_size": "30", 2024-04-24T01:55:27.780 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-04-24T01:55:27.780 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-04-24T01:55:27.780 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-04-24T01:55:27.780 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-04-24T01:55:27.780 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-04-24T01:55:27.780 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-04-24T01:55:27.780 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-04-24T01:55:27.781 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_tier_default_cache_mode": "writeback", 2024-04-24T01:55:27.781 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-04-24T01:55:27.781 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-04-24T01:55:27.781 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_tracing": "false", 2024-04-24T01:55:27.781 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_use_stale_snap": "false", 2024-04-24T01:55:27.781 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-04-24T01:55:27.781 INFO:teuthology.orchestra.run.smithi079.stdout: "osdc_blkin_trace_all": "false", 2024-04-24T01:55:27.781 INFO:teuthology.orchestra.run.smithi079.stdout: "paxos_kill_at": "0", 2024-04-24T01:55:27.781 INFO:teuthology.orchestra.run.smithi079.stdout: "paxos_max_join_drift": "10", 2024-04-24T01:55:27.781 INFO:teuthology.orchestra.run.smithi079.stdout: "paxos_min": "500", 2024-04-24T01:55:27.781 INFO:teuthology.orchestra.run.smithi079.stdout: "paxos_min_wait": "0.050000", 2024-04-24T01:55:27.781 INFO:teuthology.orchestra.run.smithi079.stdout: "paxos_propose_interval": "1.000000", 2024-04-24T01:55:27.781 INFO:teuthology.orchestra.run.smithi079.stdout: "paxos_service_trim_max": "500", 2024-04-24T01:55:27.781 INFO:teuthology.orchestra.run.smithi079.stdout: "paxos_service_trim_max_multiplier": "20", 2024-04-24T01:55:27.782 INFO:teuthology.orchestra.run.smithi079.stdout: "paxos_service_trim_min": "10", 2024-04-24T01:55:27.782 INFO:teuthology.orchestra.run.smithi079.stdout: "paxos_stash_full_interval": "25", 2024-04-24T01:55:27.782 INFO:teuthology.orchestra.run.smithi079.stdout: "paxos_trim_max": "500", 2024-04-24T01:55:27.782 INFO:teuthology.orchestra.run.smithi079.stdout: "paxos_trim_min": "250", 2024-04-24T01:55:27.782 INFO:teuthology.orchestra.run.smithi079.stdout: "perf": "true", 2024-04-24T01:55:27.782 INFO:teuthology.orchestra.run.smithi079.stdout: "pid_file": "/var/run/ceph/ceph-mon.a.pid", 2024-04-24T01:55:27.782 INFO:teuthology.orchestra.run.smithi079.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-04-24T01:55:27.782 INFO:teuthology.orchestra.run.smithi079.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-04-24T01:55:27.782 INFO:teuthology.orchestra.run.smithi079.stdout: "public_addr": "-", 2024-04-24T01:55:27.782 INFO:teuthology.orchestra.run.smithi079.stdout: "public_addrv": "", 2024-04-24T01:55:27.782 INFO:teuthology.orchestra.run.smithi079.stdout: "public_bind_addr": "-", 2024-04-24T01:55:27.782 INFO:teuthology.orchestra.run.smithi079.stdout: "public_network": "", 2024-04-24T01:55:27.782 INFO:teuthology.orchestra.run.smithi079.stdout: "public_network_interface": "", 2024-04-24T01:55:27.782 INFO:teuthology.orchestra.run.smithi079.stdout: "qat_compressor_enabled": "false", 2024-04-24T01:55:27.783 INFO:teuthology.orchestra.run.smithi079.stdout: "qat_compressor_session_max_number": "256", 2024-04-24T01:55:27.783 INFO:teuthology.orchestra.run.smithi079.stdout: "rados_mon_op_timeout": "0", 2024-04-24T01:55:27.783 INFO:teuthology.orchestra.run.smithi079.stdout: "rados_osd_op_timeout": "0", 2024-04-24T01:55:27.783 INFO:teuthology.orchestra.run.smithi079.stdout: "rados_tracing": "false", 2024-04-24T01:55:27.783 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_atime_update_interval": "60", 2024-04-24T01:55:27.783 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-04-24T01:55:27.783 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_balance_parent_reads": "false", 2024-04-24T01:55:27.783 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_balance_snap_reads": "false", 2024-04-24T01:55:27.783 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_blkin_trace_all": "false", 2024-04-24T01:55:27.783 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_blocklist_expire_seconds": "0", 2024-04-24T01:55:27.783 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_blocklist_on_break_lock": "true", 2024-04-24T01:55:27.783 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_cache": "true", 2024-04-24T01:55:27.783 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_cache_block_writes_upfront": "false", 2024-04-24T01:55:27.783 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_cache_max_dirty": "25165824", 2024-04-24T01:55:27.783 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-04-24T01:55:27.784 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_cache_max_dirty_object": "0", 2024-04-24T01:55:27.784 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_cache_policy": "writearound", 2024-04-24T01:55:27.784 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_cache_size": "33554432", 2024-04-24T01:55:27.784 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_cache_target_dirty": "16777216", 2024-04-24T01:55:27.784 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-04-24T01:55:27.784 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_clone_copy_on_read": "false", 2024-04-24T01:55:27.784 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_compression_hint": "none", 2024-04-24T01:55:27.784 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_concurrent_management_ops": "10", 2024-04-24T01:55:27.784 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-04-24T01:55:27.784 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_default_clone_format": "auto", 2024-04-24T01:55:27.784 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_default_data_pool": "", 2024-04-24T01:55:27.784 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_default_features": "61", 2024-04-24T01:55:27.784 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_default_format": "2", 2024-04-24T01:55:27.784 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_default_map_options": "", 2024-04-24T01:55:27.785 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_default_order": "22", 2024-04-24T01:55:27.785 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_default_pool": "rbd", 2024-04-24T01:55:27.785 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-04-24T01:55:27.785 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_default_stripe_count": "0", 2024-04-24T01:55:27.785 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_default_stripe_unit": "0", 2024-04-24T01:55:27.785 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_disable_zero_copy_writes": "true", 2024-04-24T01:55:27.785 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_discard_granularity_bytes": "65536", 2024-04-24T01:55:27.785 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-04-24T01:55:27.785 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_enable_alloc_hint": "true", 2024-04-24T01:55:27.785 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-04-24T01:55:27.785 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_io_scheduler": "simple", 2024-04-24T01:55:27.785 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-04-24T01:55:27.785 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_journal_commit_age": "5.000000", 2024-04-24T01:55:27.785 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-04-24T01:55:27.785 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-04-24T01:55:27.786 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-04-24T01:55:27.786 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-04-24T01:55:27.786 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_journal_object_flush_interval": "0", 2024-04-24T01:55:27.786 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-04-24T01:55:27.786 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-04-24T01:55:27.786 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_journal_order": "24", 2024-04-24T01:55:27.786 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_journal_pool": "", 2024-04-24T01:55:27.786 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_journal_splay_width": "4", 2024-04-24T01:55:27.786 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_localize_parent_reads": "false", 2024-04-24T01:55:27.786 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_localize_snap_reads": "false", 2024-04-24T01:55:27.786 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-04-24T01:55:27.786 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-04-24T01:55:27.786 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-04-24T01:55:27.786 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-04-24T01:55:27.787 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-04-24T01:55:27.787 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-04-24T01:55:27.787 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_mirror_image_policy_type": "simple", 2024-04-24T01:55:27.787 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-04-24T01:55:27.787 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-04-24T01:55:27.787 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-04-24T01:55:27.787 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-04-24T01:55:27.787 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-04-24T01:55:27.787 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-04-24T01:55:27.787 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-04-24T01:55:27.787 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_mirror_memory_autotune": "true", 2024-04-24T01:55:27.787 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_mirror_memory_base": "805306368", 2024-04-24T01:55:27.787 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-04-24T01:55:27.787 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-04-24T01:55:27.788 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-04-24T01:55:27.788 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-04-24T01:55:27.788 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_mirror_memory_target": "4294967296", 2024-04-24T01:55:27.788 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-04-24T01:55:27.788 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-04-24T01:55:27.788 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-04-24T01:55:27.788 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_mirroring_delete_delay": "0", 2024-04-24T01:55:27.788 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-04-24T01:55:27.788 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_mirroring_replay_delay": "0", 2024-04-24T01:55:27.788 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-04-24T01:55:27.788 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-04-24T01:55:27.788 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_move_to_trash_on_remove": "false", 2024-04-24T01:55:27.788 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-04-24T01:55:27.788 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_mtime_update_interval": "60", 2024-04-24T01:55:27.789 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_non_blocking_aio": "true", 2024-04-24T01:55:27.789 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_op_thread_timeout": "60", 2024-04-24T01:55:27.789 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_op_threads": "1", 2024-04-24T01:55:27.789 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_parent_cache_enabled": "false", 2024-04-24T01:55:27.789 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_persistent_cache_mode": "disabled", 2024-04-24T01:55:27.789 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_persistent_cache_path": "/tmp", 2024-04-24T01:55:27.789 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_persistent_cache_size": "1073741824", 2024-04-24T01:55:27.789 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_plugins": "", 2024-04-24T01:55:27.789 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_qos_bps_burst": "0", 2024-04-24T01:55:27.789 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-04-24T01:55:27.789 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_qos_bps_limit": "0", 2024-04-24T01:55:27.789 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_qos_exclude_ops": "0", 2024-04-24T01:55:27.789 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_qos_iops_burst": "0", 2024-04-24T01:55:27.789 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-04-24T01:55:27.790 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_qos_iops_limit": "0", 2024-04-24T01:55:27.790 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_qos_read_bps_burst": "0", 2024-04-24T01:55:27.790 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-04-24T01:55:27.790 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_qos_read_bps_limit": "0", 2024-04-24T01:55:27.790 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_qos_read_iops_burst": "0", 2024-04-24T01:55:27.790 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-04-24T01:55:27.790 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_qos_read_iops_limit": "0", 2024-04-24T01:55:27.790 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_qos_schedule_tick_min": "50", 2024-04-24T01:55:27.790 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_qos_write_bps_burst": "0", 2024-04-24T01:55:27.790 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-04-24T01:55:27.790 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_qos_write_bps_limit": "0", 2024-04-24T01:55:27.790 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_qos_write_iops_burst": "0", 2024-04-24T01:55:27.790 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-04-24T01:55:27.790 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_qos_write_iops_limit": "0", 2024-04-24T01:55:27.790 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_quiesce_notification_attempts": "10", 2024-04-24T01:55:27.791 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_read_from_replica_policy": "default", 2024-04-24T01:55:27.791 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-04-24T01:55:27.791 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_readahead_max_bytes": "524288", 2024-04-24T01:55:27.791 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_readahead_trigger_requests": "10", 2024-04-24T01:55:27.791 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_request_timed_out_seconds": "30", 2024-04-24T01:55:27.791 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_skip_partial_discard": "true", 2024-04-24T01:55:27.791 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-04-24T01:55:27.791 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_tracing": "false", 2024-04-24T01:55:27.791 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_validate_names": "true", 2024-04-24T01:55:27.791 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_validate_pool": "true", 2024-04-24T01:55:27.791 INFO:teuthology.orchestra.run.smithi079.stdout: "restapi_base_url": "", 2024-04-24T01:55:27.791 INFO:teuthology.orchestra.run.smithi079.stdout: "restapi_log_level": "", 2024-04-24T01:55:27.791 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_acl_grants_max_num": "100", 2024-04-24T01:55:27.791 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_admin_entry": "admin", 2024-04-24T01:55:27.792 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-04-24T01:55:27.792 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_backend_store": "rados", 2024-04-24T01:55:27.792 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_barbican_url": "", 2024-04-24T01:55:27.792 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_beast_enable_async": "true", 2024-04-24T01:55:27.792 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_bucket_counters_cache": "false", 2024-04-24T01:55:27.792 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-04-24T01:55:27.792 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-04-24T01:55:27.792 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-04-24T01:55:27.792 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_bucket_index_max_aio": "128", 2024-04-24T01:55:27.792 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-04-24T01:55:27.792 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-04-24T01:55:27.792 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_bucket_quota_ttl": "600", 2024-04-24T01:55:27.792 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-04-24T01:55:27.792 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_cache_enabled": "true", 2024-04-24T01:55:27.792 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_cache_expiry_interval": "900", 2024-04-24T01:55:27.793 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_cache_lru_size": "10000", 2024-04-24T01:55:27.793 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_config_store": "rados", 2024-04-24T01:55:27.793 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_content_length_compat": "false", 2024-04-24T01:55:27.793 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_copy_obj_progress": "true", 2024-04-24T01:55:27.793 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-04-24T01:55:27.793 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_cors_rules_max_num": "100", 2024-04-24T01:55:27.793 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_cross_domain_policy": "", 2024-04-24T01:55:27.793 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_crypt_default_encryption_key": "", 2024-04-24T01:55:27.793 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_crypt_kmip_addr": "", 2024-04-24T01:55:27.793 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_crypt_kmip_ca_path": "", 2024-04-24T01:55:27.793 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_crypt_kmip_client_cert": "", 2024-04-24T01:55:27.793 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_crypt_kmip_client_key": "", 2024-04-24T01:55:27.793 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-04-24T01:55:27.793 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_crypt_kmip_password": "", 2024-04-24T01:55:27.794 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-04-24T01:55:27.794 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_crypt_kmip_username": "", 2024-04-24T01:55:27.794 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_crypt_require_ssl": "true", 2024-04-24T01:55:27.794 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-04-24T01:55:27.794 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-04-24T01:55:27.794 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-04-24T01:55:27.794 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-04-24T01:55:27.794 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-04-24T01:55:27.794 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-04-24T01:55:27.794 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-04-24T01:55:27.794 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-04-24T01:55:27.794 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-04-24T01:55:27.794 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-04-24T01:55:27.794 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-04-24T01:55:27.795 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-04-24T01:55:27.795 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-04-24T01:55:27.795 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-04-24T01:55:27.795 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_crypt_suppress_logs": "true", 2024-04-24T01:55:27.795 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_crypt_vault_addr": "", 2024-04-24T01:55:27.795 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_crypt_vault_auth": "token", 2024-04-24T01:55:27.795 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_crypt_vault_namespace": "", 2024-04-24T01:55:27.795 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_crypt_vault_prefix": "", 2024-04-24T01:55:27.795 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-04-24T01:55:27.795 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-04-24T01:55:27.795 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-04-24T01:55:27.795 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-04-24T01:55:27.795 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_crypt_vault_token_file": "", 2024-04-24T01:55:27.795 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-04-24T01:55:27.796 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_curl_buffersize": "524288", 2024-04-24T01:55:27.796 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_curl_low_speed_limit": "1024", 2024-04-24T01:55:27.796 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_curl_low_speed_time": "300", 2024-04-24T01:55:27.796 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_curl_tcp_keepalive": "0", 2024-04-24T01:55:27.796 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-04-24T01:55:27.796 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-04-24T01:55:27.796 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-04-24T01:55:27.796 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-04-24T01:55:27.796 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-04-24T01:55:27.796 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_d3n_l1_fadvise": "4", 2024-04-24T01:55:27.796 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-04-24T01:55:27.796 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-04-24T01:55:27.796 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-04-24T01:55:27.796 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_d4n_host": "127.0.0.1", 2024-04-24T01:55:27.797 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_d4n_l1_datacache_persistent_path": "/tmp/rgw_d4n_datacache/", 2024-04-24T01:55:27.797 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_d4n_l1_datacache_size": "1073741824", 2024-04-24T01:55:27.797 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_d4n_l1_evict_cache_on_start": "true", 2024-04-24T01:55:27.797 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_d4n_l1_fadvise": "4", 2024-04-24T01:55:27.797 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_d4n_libaio_aio_num": "64", 2024-04-24T01:55:27.797 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_d4n_libaio_aio_threads": "20", 2024-04-24T01:55:27.797 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_d4n_port": "6379", 2024-04-24T01:55:27.797 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-04-24T01:55:27.797 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_data_log_changes_size": "1000", 2024-04-24T01:55:27.797 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_data_log_num_shards": "128", 2024-04-24T01:55:27.797 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-04-24T01:55:27.797 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_data_log_window": "30", 2024-04-24T01:55:27.797 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_data_notify_interval_msec": "0", 2024-04-24T01:55:27.797 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_data_sync_poll_interval": "20", 2024-04-24T01:55:27.797 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_data_sync_spawn_window": "20", 2024-04-24T01:55:27.798 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-04-24T01:55:27.798 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-04-24T01:55:27.798 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-04-24T01:55:27.798 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_default_data_log_backing": "fifo", 2024-04-24T01:55:27.798 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-04-24T01:55:27.798 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_default_region_info_oid": "default.region", 2024-04-24T01:55:27.798 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-04-24T01:55:27.798 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-04-24T01:55:27.798 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_defer_to_bucket_acls": "", 2024-04-24T01:55:27.798 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-04-24T01:55:27.798 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-04-24T01:55:27.798 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-04-24T01:55:27.798 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-04-24T01:55:27.799 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-04-24T01:55:27.799 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-04-24T01:55:27.799 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-04-24T01:55:27.799 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-04-24T01:55:27.799 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_dmclock_data_res": "500.000000", 2024-04-24T01:55:27.799 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-04-24T01:55:27.799 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-04-24T01:55:27.799 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-04-24T01:55:27.799 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-04-24T01:55:27.799 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_dns_name": "", 2024-04-24T01:55:27.799 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_dns_s3website_name": "", 2024-04-24T01:55:27.799 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_dynamic_resharding": "true", 2024-04-24T01:55:27.799 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-04-24T01:55:27.799 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_enable_gc_threads": "true", 2024-04-24T01:55:27.799 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_enable_lc_threads": "true", 2024-04-24T01:55:27.800 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_enable_ops_log": "false", 2024-04-24T01:55:27.800 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_enable_quota_threads": "true", 2024-04-24T01:55:27.800 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_enable_static_website": "false", 2024-04-24T01:55:27.800 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_enable_usage_log": "false", 2024-04-24T01:55:27.800 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_enforce_swift_acls": "true", 2024-04-24T01:55:27.800 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_exit_timeout_secs": "120", 2024-04-24T01:55:27.800 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_expose_bucket": "false", 2024-04-24T01:55:27.800 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_extended_http_attrs": "", 2024-04-24T01:55:27.800 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_filter": "none", 2024-04-24T01:55:27.800 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-04-24T01:55:27.800 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_frontends": "beast port=7480", 2024-04-24T01:55:27.800 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_gc_max_concurrent_io": "10", 2024-04-24T01:55:27.800 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_gc_max_deferred": "50", 2024-04-24T01:55:27.800 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-04-24T01:55:27.801 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_gc_max_objs": "32", 2024-04-24T01:55:27.801 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_gc_max_queue_size": "134213632", 2024-04-24T01:55:27.801 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_gc_max_trim_chunk": "16", 2024-04-24T01:55:27.801 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_gc_obj_min_wait": "7200", 2024-04-24T01:55:27.801 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_gc_processor_max_time": "3600", 2024-04-24T01:55:27.801 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_gc_processor_period": "3600", 2024-04-24T01:55:27.801 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-04-24T01:55:27.801 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_get_obj_window_size": "16777216", 2024-04-24T01:55:27.801 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_healthcheck_disabling_path": "", 2024-04-24T01:55:27.801 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_ignore_get_invalid_range": "false", 2024-04-24T01:55:27.801 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_init_timeout": "300", 2024-04-24T01:55:27.801 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-04-24T01:55:27.801 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-04-24T01:55:27.801 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_kafka_connection_idle": "300", 2024-04-24T01:55:27.802 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_kafka_message_timeout": "5000", 2024-04-24T01:55:27.802 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_kafka_sleep_timeout": "10", 2024-04-24T01:55:27.802 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-04-24T01:55:27.802 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-04-24T01:55:27.802 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-04-24T01:55:27.802 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_keystone_admin_domain": "", 2024-04-24T01:55:27.802 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_keystone_admin_password": "", 2024-04-24T01:55:27.802 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_keystone_admin_password_path": "", 2024-04-24T01:55:27.802 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_keystone_admin_project": "", 2024-04-24T01:55:27.802 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_keystone_admin_tenant": "", 2024-04-24T01:55:27.802 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_keystone_admin_token": "", 2024-04-24T01:55:27.802 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_keystone_admin_token_path": "", 2024-04-24T01:55:27.802 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_keystone_admin_user": "", 2024-04-24T01:55:27.802 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_keystone_api_version": "2", 2024-04-24T01:55:27.802 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_keystone_barbican_domain": "", 2024-04-24T01:55:27.803 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_keystone_barbican_password": "", 2024-04-24T01:55:27.803 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_keystone_barbican_project": "", 2024-04-24T01:55:27.803 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_keystone_barbican_tenant": "", 2024-04-24T01:55:27.803 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_keystone_barbican_user": "", 2024-04-24T01:55:27.803 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-04-24T01:55:27.803 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_keystone_implicit_tenants": "false", 2024-04-24T01:55:27.803 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-04-24T01:55:27.803 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_keystone_service_token_enabled": "false", 2024-04-24T01:55:27.803 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_keystone_token_cache_size": "10000", 2024-04-24T01:55:27.803 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_keystone_url": "", 2024-04-24T01:55:27.803 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_keystone_verify_ssl": "true", 2024-04-24T01:55:27.803 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_lc_debug_interval": "-1", 2024-04-24T01:55:27.803 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_lc_lock_max_time": "90", 2024-04-24T01:55:27.803 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_lc_max_objs": "32", 2024-04-24T01:55:27.803 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_lc_max_rules": "1000", 2024-04-24T01:55:27.804 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_lc_max_worker": "3", 2024-04-24T01:55:27.804 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_lc_max_wp_worker": "3", 2024-04-24T01:55:27.804 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_lc_thread_delay": "0", 2024-04-24T01:55:27.804 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-04-24T01:55:27.804 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_ldap_dnattr": "uid", 2024-04-24T01:55:27.804 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-04-24T01:55:27.805 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_ldap_searchfilter": "", 2024-04-24T01:55:27.805 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-04-24T01:55:27.805 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_ldap_uri": "", 2024-04-24T01:55:27.805 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_lfuda_sync_frequency": "60", 2024-04-24T01:55:27.805 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-04-24T01:55:27.805 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-04-24T01:55:27.805 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-04-24T01:55:27.805 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_local_cache_address": "127.0.0.1:6379", 2024-04-24T01:55:27.805 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_log_http_headers": "", 2024-04-24T01:55:27.806 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_log_nonexistent_bucket": "false", 2024-04-24T01:55:27.806 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-04-24T01:55:27.806 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_log_object_name_utc": "false", 2024-04-24T01:55:27.806 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-04-24T01:55:27.806 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-04-24T01:55:27.806 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_max_attr_name_len": "0", 2024-04-24T01:55:27.806 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_max_attr_size": "0", 2024-04-24T01:55:27.806 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_max_attrs_num_in_req": "0", 2024-04-24T01:55:27.806 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_max_chunk_size": "4194304", 2024-04-24T01:55:27.806 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_max_concurrent_requests": "1024", 2024-04-24T01:55:27.806 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-04-24T01:55:27.806 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_max_dynamic_shards": "1999", 2024-04-24T01:55:27.806 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_max_listing_results": "1000", 2024-04-24T01:55:27.807 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_max_notify_retries": "10", 2024-04-24T01:55:27.807 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_max_objs_per_shard": "100000", 2024-04-24T01:55:27.807 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_max_put_param_size": "1048576", 2024-04-24T01:55:27.807 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_max_put_size": "5368709120", 2024-04-24T01:55:27.807 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_max_slo_entries": "1000", 2024-04-24T01:55:27.807 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_md_log_max_shards": "64", 2024-04-24T01:55:27.807 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_md_notify_interval_msec": "200", 2024-04-24T01:55:27.807 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_meta_sync_poll_interval": "20", 2024-04-24T01:55:27.807 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_meta_sync_spawn_window": "20", 2024-04-24T01:55:27.807 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-04-24T01:55:27.807 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_mp_lock_max_time": "600", 2024-04-24T01:55:27.807 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-04-24T01:55:27.807 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_multipart_min_part_size": "5242880", 2024-04-24T01:55:27.807 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-04-24T01:55:27.808 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-04-24T01:55:27.808 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_nfs_fhcache_size": "2017", 2024-04-24T01:55:27.808 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-04-24T01:55:27.808 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-04-24T01:55:27.808 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_nfs_lru_lanes": "5", 2024-04-24T01:55:27.808 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_nfs_max_gc": "300", 2024-04-24T01:55:27.808 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-04-24T01:55:27.808 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_nfs_run_gc_threads": "false", 2024-04-24T01:55:27.808 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_nfs_run_lc_threads": "false", 2024-04-24T01:55:27.808 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_nfs_run_quota_threads": "false", 2024-04-24T01:55:27.808 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_nfs_run_sync_thread": "false", 2024-04-24T01:55:27.808 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-04-24T01:55:27.808 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-04-24T01:55:27.809 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_num_async_rados_threads": "32", 2024-04-24T01:55:27.809 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_num_control_oids": "8", 2024-04-24T01:55:27.809 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_numa_node": "-1", 2024-04-24T01:55:27.809 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_obj_stripe_size": "4194304", 2024-04-24T01:55:27.809 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-04-24T01:55:27.809 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_objexp_chunk_size": "100", 2024-04-24T01:55:27.809 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_objexp_gc_interval": "600", 2024-04-24T01:55:27.809 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_objexp_hints_num_shards": "127", 2024-04-24T01:55:27.809 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-04-24T01:55:27.809 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-04-24T01:55:27.809 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_op_thread_timeout": "600", 2024-04-24T01:55:27.809 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_op_tracing": "false", 2024-04-24T01:55:27.809 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_opa_token": "", 2024-04-24T01:55:27.809 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_opa_url": "", 2024-04-24T01:55:27.809 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_opa_verify_ssl": "true", 2024-04-24T01:55:27.810 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-04-24T01:55:27.810 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-04-24T01:55:27.810 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_ops_log_rados": "false", 2024-04-24T01:55:27.810 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_ops_log_socket_path": "", 2024-04-24T01:55:27.810 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-04-24T01:55:27.810 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-04-24T01:55:27.810 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-04-24T01:55:27.810 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_period_push_interval": "2.000000", 2024-04-24T01:55:27.810 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_period_push_interval_max": "30.000000", 2024-04-24T01:55:27.810 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_period_root_pool": ".rgw.root", 2024-04-24T01:55:27.810 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-04-24T01:55:27.810 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-04-24T01:55:27.810 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_posix_cache_lanes": "3", 2024-04-24T01:55:27.811 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-04-24T01:55:27.811 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_posix_cache_max_buckets": "100", 2024-04-24T01:55:27.811 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_posix_cache_partitions": "3", 2024-04-24T01:55:27.811 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-04-24T01:55:27.811 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_print_continue": "true", 2024-04-24T01:55:27.811 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_print_prohibited_content_length": "false", 2024-04-24T01:55:27.811 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-04-24T01:55:27.811 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-04-24T01:55:27.811 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-04-24T01:55:27.811 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-04-24T01:55:27.811 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_rados_tracing": "false", 2024-04-24T01:55:27.811 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_realm": "", 2024-04-24T01:55:27.811 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_realm_id": "", 2024-04-24T01:55:27.812 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-04-24T01:55:27.812 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_region": "", 2024-04-24T01:55:27.812 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_region_root_pool": ".rgw.root", 2024-04-24T01:55:27.812 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_relaxed_region_enforcement": "false", 2024-04-24T01:55:27.812 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-04-24T01:55:27.812 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-04-24T01:55:27.812 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_request_uri": "", 2024-04-24T01:55:27.812 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_reshard_batch_size": "64", 2024-04-24T01:55:27.812 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-04-24T01:55:27.812 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_reshard_max_aio": "128", 2024-04-24T01:55:27.812 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_reshard_num_logs": "16", 2024-04-24T01:55:27.812 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_reshard_thread_interval": "600", 2024-04-24T01:55:27.812 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_resolve_cname": "false", 2024-04-24T01:55:27.812 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_rest_getusage_op_compat": "false", 2024-04-24T01:55:27.812 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_run_sync_thread": "true", 2024-04-24T01:55:27.813 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_s3_auth_disable_signature_url": "false", 2024-04-24T01:55:27.813 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-04-24T01:55:27.813 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_s3_auth_use_keystone": "false", 2024-04-24T01:55:27.813 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_s3_auth_use_ldap": "false", 2024-04-24T01:55:27.813 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_s3_auth_use_rados": "true", 2024-04-24T01:55:27.813 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_s3_auth_use_sts": "false", 2024-04-24T01:55:27.813 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-04-24T01:55:27.813 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_s3_success_create_obj_status": "0", 2024-04-24T01:55:27.813 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-04-24T01:55:27.813 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_scheduler_type": "throttler", 2024-04-24T01:55:27.813 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_script_uri": "", 2024-04-24T01:55:27.813 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_service_provider_name": "", 2024-04-24T01:55:27.813 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-04-24T01:55:27.813 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_sts_client_id": "", 2024-04-24T01:55:27.813 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_sts_client_secret": "", 2024-04-24T01:55:27.814 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_sts_entry": "sts", 2024-04-24T01:55:27.814 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_sts_key": "sts", 2024-04-24T01:55:27.814 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_sts_max_session_duration": "43200", 2024-04-24T01:55:27.814 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_sts_min_session_duration": "900", 2024-04-24T01:55:27.814 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_sts_token_introspection_url": "", 2024-04-24T01:55:27.814 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_swift_account_in_url": "false", 2024-04-24T01:55:27.814 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_swift_auth_entry": "auth", 2024-04-24T01:55:27.814 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_swift_auth_url": "", 2024-04-24T01:55:27.814 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_swift_custom_header": "", 2024-04-24T01:55:27.814 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_swift_enforce_content_length": "false", 2024-04-24T01:55:27.814 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_swift_need_stats": "true", 2024-04-24T01:55:27.814 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_swift_tenant_name": "", 2024-04-24T01:55:27.814 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_swift_token_expiration": "86400", 2024-04-24T01:55:27.814 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_swift_url": "", 2024-04-24T01:55:27.815 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_swift_url_prefix": "swift", 2024-04-24T01:55:27.815 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_swift_versioning_enabled": "false", 2024-04-24T01:55:27.815 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-04-24T01:55:27.815 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-04-24T01:55:27.815 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_sync_lease_period": "120", 2024-04-24T01:55:27.815 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-04-24T01:55:27.815 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_sync_log_trim_interval": "1200", 2024-04-24T01:55:27.815 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-04-24T01:55:27.815 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-04-24T01:55:27.815 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-04-24T01:55:27.815 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_sync_obj_etag_verify": "false", 2024-04-24T01:55:27.815 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_sync_trace_history_size": "4096", 2024-04-24T01:55:27.815 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-04-24T01:55:27.815 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-04-24T01:55:27.816 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_thread_pool_size": "512", 2024-04-24T01:55:27.816 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_topic_persistency_max_retries": "0", 2024-04-24T01:55:27.816 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-04-24T01:55:27.816 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-04-24T01:55:27.816 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_topic_require_publish_policy": "false", 2024-04-24T01:55:27.816 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_torrent_comment": "", 2024-04-24T01:55:27.816 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_torrent_createby": "", 2024-04-24T01:55:27.816 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_torrent_encoding": "", 2024-04-24T01:55:27.816 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_torrent_flag": "false", 2024-04-24T01:55:27.816 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_torrent_max_size": "5368709120", 2024-04-24T01:55:27.816 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_torrent_origin": "", 2024-04-24T01:55:27.816 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_torrent_sha_unit": "524288", 2024-04-24T01:55:27.816 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_torrent_tracker": "", 2024-04-24T01:55:27.816 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_trust_forwarded_https": "false", 2024-04-24T01:55:27.816 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-04-24T01:55:27.817 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_usage_log_tick_interval": "30", 2024-04-24T01:55:27.817 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_usage_max_shards": "32", 2024-04-24T01:55:27.817 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_usage_max_user_shards": "1", 2024-04-24T01:55:27.817 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_use_opa_authz": "false", 2024-04-24T01:55:27.817 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_user_counters_cache": "false", 2024-04-24T01:55:27.817 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_user_counters_cache_size": "10000", 2024-04-24T01:55:27.817 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-04-24T01:55:27.817 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_user_default_quota_max_size": "-1", 2024-04-24T01:55:27.817 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_user_max_buckets": "1000", 2024-04-24T01:55:27.817 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_user_policies_max_num": "100", 2024-04-24T01:55:27.817 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-04-24T01:55:27.817 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-04-24T01:55:27.817 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_user_quota_sync_interval": "86400", 2024-04-24T01:55:27.817 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-04-24T01:55:27.817 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_user_unique_email": "true", 2024-04-24T01:55:27.818 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_verify_ssl": "true", 2024-04-24T01:55:27.818 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_website_routing_rules_max_num": "50", 2024-04-24T01:55:27.818 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_zone": "", 2024-04-24T01:55:27.818 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_zone_id": "", 2024-04-24T01:55:27.818 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-04-24T01:55:27.818 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_zonegroup": "", 2024-04-24T01:55:27.818 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_zonegroup_id": "", 2024-04-24T01:55:27.818 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-04-24T01:55:27.818 INFO:teuthology.orchestra.run.smithi079.stdout: "rgwlc_auto_session_clear": "true", 2024-04-24T01:55:27.818 INFO:teuthology.orchestra.run.smithi079.stdout: "rgwlc_skip_bucket_step": "false", 2024-04-24T01:55:27.818 INFO:teuthology.orchestra.run.smithi079.stdout: "rocksdb_block_size": "4096", 2024-04-24T01:55:27.818 INFO:teuthology.orchestra.run.smithi079.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-04-24T01:55:27.818 INFO:teuthology.orchestra.run.smithi079.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-04-24T01:55:27.818 INFO:teuthology.orchestra.run.smithi079.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-04-24T01:55:27.818 INFO:teuthology.orchestra.run.smithi079.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-04-24T01:55:27.819 INFO:teuthology.orchestra.run.smithi079.stdout: "rocksdb_cache_shard_bits": "4", 2024-04-24T01:55:27.819 INFO:teuthology.orchestra.run.smithi079.stdout: "rocksdb_cache_size": "536870912", 2024-04-24T01:55:27.819 INFO:teuthology.orchestra.run.smithi079.stdout: "rocksdb_cache_type": "binned_lru", 2024-04-24T01:55:27.819 INFO:teuthology.orchestra.run.smithi079.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-04-24T01:55:27.819 INFO:teuthology.orchestra.run.smithi079.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-04-24T01:55:27.819 INFO:teuthology.orchestra.run.smithi079.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-04-24T01:55:27.819 INFO:teuthology.orchestra.run.smithi079.stdout: "rocksdb_collect_compaction_stats": "false", 2024-04-24T01:55:27.819 INFO:teuthology.orchestra.run.smithi079.stdout: "rocksdb_collect_extended_stats": "false", 2024-04-24T01:55:27.819 INFO:teuthology.orchestra.run.smithi079.stdout: "rocksdb_collect_memory_stats": "false", 2024-04-24T01:55:27.819 INFO:teuthology.orchestra.run.smithi079.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-04-24T01:55:27.819 INFO:teuthology.orchestra.run.smithi079.stdout: "rocksdb_index_type": "binary_search", 2024-04-24T01:55:27.819 INFO:teuthology.orchestra.run.smithi079.stdout: "rocksdb_log_to_ceph_log": "true", 2024-04-24T01:55:27.819 INFO:teuthology.orchestra.run.smithi079.stdout: "rocksdb_metadata_block_size": "4096", 2024-04-24T01:55:27.819 INFO:teuthology.orchestra.run.smithi079.stdout: "rocksdb_partition_filters": "false", 2024-04-24T01:55:27.819 INFO:teuthology.orchestra.run.smithi079.stdout: "rocksdb_perf": "false", 2024-04-24T01:55:27.820 INFO:teuthology.orchestra.run.smithi079.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-04-24T01:55:27.820 INFO:teuthology.orchestra.run.smithi079.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-04-24T01:55:27.820 INFO:teuthology.orchestra.run.smithi079.stdout: "rotating_keys_renewal_timeout": "10", 2024-04-24T01:55:27.820 INFO:teuthology.orchestra.run.smithi079.stdout: "run_dir": "/var/run/ceph", 2024-04-24T01:55:27.820 INFO:teuthology.orchestra.run.smithi079.stdout: "seastore_block_create": "true", 2024-04-24T01:55:27.820 INFO:teuthology.orchestra.run.smithi079.stdout: "seastore_cache_lru_size": "67108864", 2024-04-24T01:55:27.820 INFO:teuthology.orchestra.run.smithi079.stdout: "seastore_cbjournal_size": "5368709120", 2024-04-24T01:55:27.820 INFO:teuthology.orchestra.run.smithi079.stdout: "seastore_data_delta_based_overwrite": "0", 2024-04-24T01:55:27.820 INFO:teuthology.orchestra.run.smithi079.stdout: "seastore_default_max_object_size": "16777216", 2024-04-24T01:55:27.820 INFO:teuthology.orchestra.run.smithi079.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-04-24T01:55:27.820 INFO:teuthology.orchestra.run.smithi079.stdout: "seastore_device_size": "53687091200", 2024-04-24T01:55:27.820 INFO:teuthology.orchestra.run.smithi079.stdout: "seastore_journal_batch_capacity": "16", 2024-04-24T01:55:27.820 INFO:teuthology.orchestra.run.smithi079.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-04-24T01:55:27.820 INFO:teuthology.orchestra.run.smithi079.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-04-24T01:55:27.820 INFO:teuthology.orchestra.run.smithi079.stdout: "seastore_journal_iodepth_limit": "5", 2024-04-24T01:55:27.820 INFO:teuthology.orchestra.run.smithi079.stdout: "seastore_main_device_type": "SSD", 2024-04-24T01:55:27.821 INFO:teuthology.orchestra.run.smithi079.stdout: "seastore_max_concurrent_transactions": "8", 2024-04-24T01:55:27.821 INFO:teuthology.orchestra.run.smithi079.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-04-24T01:55:27.821 INFO:teuthology.orchestra.run.smithi079.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-04-24T01:55:27.821 INFO:teuthology.orchestra.run.smithi079.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-04-24T01:55:27.821 INFO:teuthology.orchestra.run.smithi079.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-04-24T01:55:27.821 INFO:teuthology.orchestra.run.smithi079.stdout: "seastore_segment_size": "67108864", 2024-04-24T01:55:27.821 INFO:teuthology.orchestra.run.smithi079.stdout: "set_keepcaps": "false", 2024-04-24T01:55:27.821 INFO:teuthology.orchestra.run.smithi079.stdout: "setgroup": "", 2024-04-24T01:55:27.821 INFO:teuthology.orchestra.run.smithi079.stdout: "setuser": "", 2024-04-24T01:55:27.821 INFO:teuthology.orchestra.run.smithi079.stdout: "setuser_match_path": "", 2024-04-24T01:55:27.821 INFO:teuthology.orchestra.run.smithi079.stdout: "target_max_misplaced_ratio": "0.050000", 2024-04-24T01:55:27.821 INFO:teuthology.orchestra.run.smithi079.stdout: "thp": "false", 2024-04-24T01:55:27.821 INFO:teuthology.orchestra.run.smithi079.stdout: "threadpool_default_timeout": "60", 2024-04-24T01:55:27.821 INFO:teuthology.orchestra.run.smithi079.stdout: "threadpool_empty_queue_max_wait": "2", 2024-04-24T01:55:27.822 INFO:teuthology.orchestra.run.smithi079.stdout: "throttler_perf_counter": "true" 2024-04-24T01:55:27.822 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-04-24T01:55:27.824 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-04-24T01:55:27.928 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:27.926+0000 7fac6139d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:27.932 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:27.931+0000 7fac6139d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:27.946 INFO:teuthology.orchestra.run.smithi079.stdout:{} 2024-04-24T01:55:27.946 INFO:teuthology.orchestra.run.smithi079.stderr:mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-24T01:55:27.957 DEBUG:teuthology.orchestra.run.smithi079:> type ceph-objectstore-tool 2024-04-24T01:55:27.971 INFO:teuthology.orchestra.run.smithi079.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-04-24T01:55:27.971 DEBUG:teuthology.orchestra.run.smithi103:> type ceph-objectstore-tool 2024-04-24T01:55:27.989 INFO:teuthology.orchestra.run.smithi103.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-04-24T01:55:27.989 INFO:teuthology.run_tasks:Running task rados... 2024-04-24T01:55:28.000 INFO:tasks.rados:Beginning rados... 2024-04-24T01:55:28.000 INFO:tasks.rados:config is {'clients': ['client.0'], 'objects': 50, 'op_weights': {'copy_from': 0, 'delete': 50, 'read': 100, 'rollback': 0, 'snap_create': 50, 'snap_remove': 50, 'write': 100}, 'ops': 4000} 2024-04-24T01:55:28.000 INFO:tasks.rados:overrides is {'admin_socket': {'branch': 'ci-yingxin-crimson-load-obc-new-1'}, 'ceph': {'conf': {'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum'], 'sha1': '424d8c4389b684b0a4d72510e1b4d398d8e6c8f4'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'flavor': 'crimson', 'sha1': '424d8c4389b684b0a4d72510e1b4d398d8e6c8f4'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'thrashosds': {'min_in': 4}, 'workunit': {'branch': 'ci-yingxin-crimson-load-obc-new-1', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': '424d8c4389b684b0a4d72510e1b4d398d8e6c8f4'}} 2024-04-24T01:55:28.000 INFO:tasks.rados:config is {'clients': ['client.0'], 'objects': 50, 'op_weights': {'copy_from': 0, 'delete': 50, 'read': 100, 'rollback': 0, 'snap_create': 50, 'snap_remove': 50, 'write': 100}, 'ops': 4000} 2024-04-24T01:55:28.000 DEBUG:teuthology.run_tasks:Unwinding manager rados 2024-04-24T01:55:28.009 INFO:tasks.rados:joining rados 2024-04-24T01:55:28.009 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-04-24T01:55:28.009 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-04-24T01:55:28.010 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-04-24T01:55:28.010 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-04-24T01:55:28.010 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-04-24T01:55:28.010 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-04-24T01:55:28.010 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T01:55:28.011 INFO:tasks.rados:clients are ['client.0', 'client.1'] 2024-04-24T01:55:28.011 INFO:tasks.rados:starting run 0 out of 1 2024-04-24T01:55:28.011 INFO:tasks.ceph.ceph_manager.ceph:creating pool_name unique_pool_0 2024-04-24T01:55:28.011 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create unique_pool_0 16 2024-04-24T01:55:28.120 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:28.118+0000 7f881902d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:28.124 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:28.118+0000 7fbca2864640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:28.124 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:28.123+0000 7f881902d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:28.126 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:28.118+0000 7f612a86d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:28.128 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:28.128+0000 7fbca2864640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:28.132 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:28.131+0000 7f612a86d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:28.164 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T01:55:28.164 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T01:55:28.164 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T01:55:28.164 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T01:55:28.164 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T01:55:28.164 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T01:55:28.164 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T01:55:28.164 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T01:55:28.164 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T01:55:28.164 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T01:55:28.164 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T01:55:28.164 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T01:55:28.167 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T01:55:28.167 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T01:55:28.167 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-04-24T01:55:28.278 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:28.273+0000 7f179b6ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:28.282 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:28.282+0000 7f179b6ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:28.305 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T01:55:28.305 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T01:55:28.306 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T01:55:28.306 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T01:55:28.306 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T01:55:28.306 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T01:55:28.306 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T01:55:28.306 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T01:55:28.306 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T01:55:28.306 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T01:55:28.306 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T01:55:28.306 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T01:55:28.309 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T01:55:28.309 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T01:55:28.309 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-04-24T01:55:28.458 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:28.453+0000 7f119a59d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:28.462 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:28.461+0000 7f119a59d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:28.480 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T01:55:28.480 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T01:55:28.480 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T01:55:28.480 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T01:55:28.480 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T01:55:28.480 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T01:55:28.480 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T01:55:28.480 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T01:55:28.480 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T01:55:28.480 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T01:55:28.481 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T01:55:28.481 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T01:55:28.483 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T01:55:28.483 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T01:55:28.483 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-04-24T01:55:28.590 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:28.585+0000 7f975a59d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:28.594 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:28.593+0000 7f975a59d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:28.610 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T01:55:28.610 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T01:55:28.610 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T01:55:28.610 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T01:55:28.610 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T01:55:28.610 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T01:55:28.611 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T01:55:28.611 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T01:55:28.611 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T01:55:28.611 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T01:55:28.611 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T01:55:28.611 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T01:55:28.613 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T01:55:28.613 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T01:55:29.173 INFO:teuthology.orchestra.run.smithi079.stderr:pool 'unique_pool_0' created 2024-04-24T01:55:29.191 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-04-24T01:55:29.327 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:29.317+0000 7f887c06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:29.330 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:29.331+0000 7f887c06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:30.181 INFO:teuthology.orchestra.run.smithi079.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-04-24T01:55:30.243 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T01:55:30.401 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:30.399+0000 7efddaa20640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:30.407 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:30.407+0000 7efddaa20640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:31.201 DEBUG:teuthology.orchestra.run.smithi079:> CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 0 --op copy_from 0 --op write_excl 50 --pool unique_pool_0 2024-04-24T01:55:31.237 INFO:tasks.rados.rados.0.smithi079.stdout:adding op weight read -> 100 2024-04-24T01:55:31.237 INFO:tasks.rados.rados.0.smithi079.stdout:adding op weight write -> 50 2024-04-24T01:55:31.237 INFO:tasks.rados.rados.0.smithi079.stdout:adding op weight delete -> 50 2024-04-24T01:55:31.237 INFO:tasks.rados.rados.0.smithi079.stdout:adding op weight snap_create -> 50 2024-04-24T01:55:31.238 INFO:tasks.rados.rados.0.smithi079.stdout:adding op weight snap_remove -> 50 2024-04-24T01:55:31.238 INFO:tasks.rados.rados.0.smithi079.stdout:adding op weight write_excl -> 50 2024-04-24T01:55:31.238 INFO:tasks.rados.rados.0.smithi079.stdout:ceph version 19.0.0-3227-g424d8c43 (424d8c4389b684b0a4d72510e1b4d398d8e6c8f4) squid (dev) 2024-04-24T01:55:31.238 INFO:tasks.rados.rados.0.smithi079.stdout:Configuration: 2024-04-24T01:55:31.238 INFO:tasks.rados.rados.0.smithi079.stdout: Number of operations: 4000 2024-04-24T01:55:31.238 INFO:tasks.rados.rados.0.smithi079.stdout: Number of objects: 50 2024-04-24T01:55:31.238 INFO:tasks.rados.rados.0.smithi079.stdout: Max in flight operations: 16 2024-04-24T01:55:31.238 INFO:tasks.rados.rados.0.smithi079.stdout: Object size (in bytes): 4000000 2024-04-24T01:55:31.238 INFO:tasks.rados.rados.0.smithi079.stdout: Write stride min: 400000 2024-04-24T01:55:31.238 INFO:tasks.rados.rados.0.smithi079.stdout: Write stride max: 800000 2024-04-24T01:55:32.191 INFO:tasks.rados.rados.0.smithi079.stdout:1: write initial oid 1 2024-04-24T01:55:32.191 INFO:tasks.rados.rados.0.smithi079.stdout:1: seq_num 0 ranges {416807=475249,1542129=474678} 2024-04-24T01:55:32.196 INFO:tasks.rados.rados.0.smithi079.stdout:1: writing smithi07929762-1 from 416807 to 892056 tid 1 2024-04-24T01:55:32.203 INFO:tasks.rados.rados.0.smithi079.stdout:1: writing smithi07929762-1 from 1542129 to 2016807 tid 2 2024-04-24T01:55:32.203 INFO:tasks.rados.rados.0.smithi079.stdout:2: write initial oid 2 2024-04-24T01:55:32.203 INFO:tasks.rados.rados.0.smithi079.stdout:2: seq_num 1 ranges {416807=475249,1542129=474678} 2024-04-24T01:55:32.208 INFO:tasks.rados.rados.0.smithi079.stdout:2: writing smithi07929762-2 from 416807 to 892056 tid 1 2024-04-24T01:55:32.215 INFO:tasks.rados.rados.0.smithi079.stdout:2: writing smithi07929762-2 from 1542129 to 2016807 tid 2 2024-04-24T01:55:32.215 INFO:tasks.rados.rados.0.smithi079.stdout:3: write initial oid 3 2024-04-24T01:55:32.215 INFO:tasks.rados.rados.0.smithi079.stdout:3: seq_num 2 ranges {433614=550498,1600611=433003} 2024-04-24T01:55:32.221 INFO:tasks.rados.rados.0.smithi079.stdout:3: writing smithi07929762-3 from 433614 to 984112 tid 1 2024-04-24T01:55:32.227 INFO:tasks.rados.rados.0.smithi079.stdout:3: writing smithi07929762-3 from 1600611 to 2033614 tid 2 2024-04-24T01:55:32.227 INFO:tasks.rados.rados.0.smithi079.stdout:4: write initial oid 4 2024-04-24T01:55:32.227 INFO:tasks.rados.rados.0.smithi079.stdout:4: seq_num 3 ranges {450421=625747,1659093=391328} 2024-04-24T01:55:32.233 INFO:tasks.rados.rados.0.smithi079.stdout:4: writing smithi07929762-4 from 450421 to 1076168 tid 1 2024-04-24T01:55:32.239 INFO:tasks.rados.rados.0.smithi079.stdout:4: writing smithi07929762-4 from 1659093 to 2050421 tid 2 2024-04-24T01:55:32.239 INFO:tasks.rados.rados.0.smithi079.stdout:5: write initial oid 5 2024-04-24T01:55:32.240 INFO:tasks.rados.rados.0.smithi079.stdout:5: seq_num 4 ranges {467228=700996,1717575=349653} 2024-04-24T01:55:32.246 INFO:tasks.rados.rados.0.smithi079.stdout:5: writing smithi07929762-5 from 467228 to 1168224 tid 1 2024-04-24T01:55:32.251 INFO:tasks.rados.rados.0.smithi079.stdout:5: writing smithi07929762-5 from 1717575 to 2067228 tid 2 2024-04-24T01:55:32.251 INFO:tasks.rados.rados.0.smithi079.stdout:6: write initial oid 6 2024-04-24T01:55:32.252 INFO:tasks.rados.rados.0.smithi079.stdout:6: seq_num 5 ranges {484035=776245,2059704=24331} 2024-04-24T01:55:32.259 INFO:tasks.rados.rados.0.smithi079.stdout:6: writing smithi07929762-6 from 484035 to 1260280 tid 1 2024-04-24T01:55:32.264 INFO:tasks.rados.rados.0.smithi079.stdout:6: writing smithi07929762-6 from 2059704 to 2084035 tid 2 2024-04-24T01:55:32.264 INFO:tasks.rados.rados.0.smithi079.stdout:7: write initial oid 7 2024-04-24T01:55:32.264 INFO:tasks.rados.rados.0.smithi079.stdout:7: seq_num 6 ranges {500842=451494,1718186=382656} 2024-04-24T01:55:32.272 INFO:tasks.rados.rados.0.smithi079.stdout:7: writing smithi07929762-7 from 500842 to 952336 tid 1 2024-04-24T01:55:32.281 INFO:tasks.rados.rados.0.smithi079.stdout:7: writing smithi07929762-7 from 1718186 to 2100842 tid 2 2024-04-24T01:55:32.281 INFO:tasks.rados.rados.0.smithi079.stdout:8: write initial oid 8 2024-04-24T01:55:32.281 INFO:tasks.rados.rados.0.smithi079.stdout:8: seq_num 7 ranges {517649=526743,1776668=340981} 2024-04-24T01:55:32.288 INFO:tasks.rados.rados.0.smithi079.stdout:8: writing smithi07929762-8 from 517649 to 1044392 tid 1 2024-04-24T01:55:32.295 INFO:tasks.rados.rados.0.smithi079.stdout:8: writing smithi07929762-8 from 1776668 to 2117649 tid 2 2024-04-24T01:55:32.295 INFO:tasks.rados.rados.0.smithi079.stdout:9: write initial oid 9 2024-04-24T01:55:32.296 INFO:tasks.rados.rados.0.smithi079.stdout:9: seq_num 8 ranges {534456=718345,1951503=182953} 2024-04-24T01:55:32.303 INFO:tasks.rados.rados.0.smithi079.stdout:9: writing smithi07929762-9 from 534456 to 1252801 tid 1 2024-04-24T01:55:32.308 INFO:tasks.rados.rados.0.smithi079.stdout:9: writing smithi07929762-9 from 1951503 to 2134456 tid 2 2024-04-24T01:55:32.308 INFO:tasks.rados.rados.0.smithi079.stdout:10: write initial oid 10 2024-04-24T01:55:32.308 INFO:tasks.rados.rados.0.smithi079.stdout:10: seq_num 9 ranges {551263=793594,1893632=257631} 2024-04-24T01:55:32.317 INFO:tasks.rados.rados.0.smithi079.stdout:10: writing smithi07929762-10 from 551263 to 1344857 tid 1 2024-04-24T01:55:32.323 INFO:tasks.rados.rados.0.smithi079.stdout:10: writing smithi07929762-10 from 1893632 to 2151263 tid 2 2024-04-24T01:55:32.323 INFO:tasks.rados.rados.0.smithi079.stdout:2: finishing write tid 1 to smithi07929762-2 2024-04-24T01:55:32.323 INFO:tasks.rados.rados.0.smithi079.stdout:6: finishing write tid 1 to smithi07929762-6 2024-04-24T01:55:32.323 INFO:tasks.rados.rados.0.smithi079.stdout:1: finishing write tid 1 to smithi07929762-1 2024-04-24T01:55:32.323 INFO:tasks.rados.rados.0.smithi079.stdout:7: finishing write tid 1 to smithi07929762-7 2024-04-24T01:55:32.324 INFO:tasks.rados.rados.0.smithi079.stdout:5: finishing write tid 1 to smithi07929762-5 2024-04-24T01:55:32.324 INFO:tasks.rados.rados.0.smithi079.stdout:11: write initial oid 11 2024-04-24T01:55:32.324 INFO:tasks.rados.rados.0.smithi079.stdout:11: seq_num 10 ranges {568070=468843,1552114=615956} 2024-04-24T01:55:32.329 INFO:tasks.rados.rados.0.smithi079.stdout:11: writing smithi07929762-11 from 568070 to 1036913 tid 1 2024-04-24T01:55:32.337 INFO:tasks.rados.rados.0.smithi079.stdout:11: writing smithi07929762-11 from 1552114 to 2168070 tid 2 2024-04-24T01:55:32.337 INFO:tasks.rados.rados.0.smithi079.stdout:12: write initial oid 12 2024-04-24T01:55:32.337 INFO:tasks.rados.rados.0.smithi079.stdout:12: seq_num 11 ranges {584877=544092,1610596=562003} 2024-04-24T01:55:32.343 INFO:tasks.rados.rados.0.smithi079.stdout:12: writing smithi07929762-12 from 584877 to 1128969 tid 1 2024-04-24T01:55:32.349 INFO:tasks.rados.rados.0.smithi079.stdout:12: writing smithi07929762-12 from 1610596 to 2172599 tid 2 2024-04-24T01:55:32.349 INFO:tasks.rados.rados.0.smithi079.stdout:13: write initial oid 13 2024-04-24T01:55:32.349 INFO:tasks.rados.rados.0.smithi079.stdout:13: seq_num 12 ranges {601684=619341,1669078=532606} 2024-04-24T01:55:32.356 INFO:tasks.rados.rados.0.smithi079.stdout:13: writing smithi07929762-13 from 601684 to 1221025 tid 1 2024-04-24T01:55:32.362 INFO:tasks.rados.rados.0.smithi079.stdout:13: writing smithi07929762-13 from 1669078 to 2201684 tid 2 2024-04-24T01:55:32.362 INFO:tasks.rados.rados.0.smithi079.stdout:14: write initial oid 14 2024-04-24T01:55:32.363 INFO:tasks.rados.rados.0.smithi079.stdout:14: seq_num 13 ranges {618491=694590,2011207=207284} 2024-04-24T01:55:32.371 INFO:tasks.rados.rados.0.smithi079.stdout:14: writing smithi07929762-14 from 618491 to 1313081 tid 1 2024-04-24T01:55:32.377 INFO:tasks.rados.rados.0.smithi079.stdout:14: writing smithi07929762-14 from 2011207 to 2218491 tid 2 2024-04-24T01:55:32.377 INFO:tasks.rados.rados.0.smithi079.stdout:15: write initial oid 15 2024-04-24T01:55:32.377 INFO:tasks.rados.rados.0.smithi079.stdout:15: seq_num 14 ranges {635298=769839,2069689=165609} 2024-04-24T01:55:32.387 INFO:tasks.rados.rados.0.smithi079.stdout:15: writing smithi07929762-15 from 635298 to 1405137 tid 1 2024-04-24T01:55:32.392 INFO:tasks.rados.rados.0.smithi079.stdout:15: writing smithi07929762-15 from 2069689 to 2235298 tid 2 2024-04-24T01:55:32.392 INFO:tasks.rados.rados.0.smithi079.stdout:16: write initial oid 16 2024-04-24T01:55:32.392 INFO:tasks.rados.rados.0.smithi079.stdout:16: seq_num 15 ranges {652105=445088,1728171=452988} 2024-04-24T01:55:32.399 INFO:tasks.rados.rados.0.smithi079.stdout:16: writing smithi07929762-16 from 652105 to 1097193 tid 1 2024-04-24T01:55:32.407 INFO:tasks.rados.rados.0.smithi079.stdout:16: writing smithi07929762-16 from 1728171 to 2181159 tid 2 2024-04-24T01:55:32.407 INFO:tasks.rados.rados.0.smithi079.stdout: waiting on 16 2024-04-24T01:55:32.407 INFO:tasks.rados.rados.0.smithi079.stdout:4: finishing write tid 1 to smithi07929762-4 2024-04-24T01:55:32.407 INFO:tasks.rados.rados.0.smithi079.stdout:2: finishing write tid 2 to smithi07929762-2 2024-04-24T01:55:32.407 INFO:tasks.rados.rados.0.smithi079.stdout:6: finishing write tid 2 to smithi07929762-6 2024-04-24T01:55:32.407 INFO:tasks.rados.rados.0.smithi079.stdout:7: finishing write tid 2 to smithi07929762-7 2024-04-24T01:55:32.407 INFO:tasks.rados.rados.0.smithi079.stdout:6: finishing write tid 3 to smithi07929762-6 2024-04-24T01:55:32.407 INFO:tasks.rados.rados.0.smithi079.stdout:1: finishing write tid 2 to smithi07929762-1 2024-04-24T01:55:32.407 INFO:tasks.rados.rados.0.smithi079.stdout:1: finishing write tid 3 to smithi07929762-1 2024-04-24T01:55:32.407 INFO:tasks.rados.rados.0.smithi079.stdout:7: finishing write tid 3 to smithi07929762-7 2024-04-24T01:55:32.407 INFO:tasks.rados.rados.0.smithi079.stdout:6: finishing write tid 4 to smithi07929762-6 2024-04-24T01:55:32.408 INFO:tasks.rados.rados.0.smithi079.stdout:6: oid 6 updating version 0 to 3 2024-04-24T01:55:32.408 INFO:tasks.rados.rados.0.smithi079.stdout:6: oid 6 version 3 is already newer than 1 2024-04-24T01:55:32.408 INFO:tasks.rados.rados.0.smithi079.stdout:6: oid 6 version 3 is already newer than 3 2024-04-24T01:55:32.408 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 6 v 3 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-04-24T01:55:32.408 INFO:tasks.rados.rados.0.smithi079.stdout:6: left oid 6 (ObjNum 5 snap 0 seq_num 5) 2024-04-24T01:55:32.408 INFO:tasks.rados.rados.0.smithi079.stdout:1: finishing write tid 4 to smithi07929762-1 2024-04-24T01:55:32.408 INFO:tasks.rados.rados.0.smithi079.stdout:1: oid 1 updating version 0 to 3 2024-04-24T01:55:32.408 INFO:tasks.rados.rados.0.smithi079.stdout:1: oid 1 version 3 is already newer than 1 2024-04-24T01:55:32.408 INFO:tasks.rados.rados.0.smithi079.stdout:1: oid 1 version 3 is already newer than 3 2024-04-24T01:55:32.408 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 1 v 3 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-04-24T01:55:32.408 INFO:tasks.rados.rados.0.smithi079.stdout:1: left oid 1 (ObjNum 0 snap 0 seq_num 0) 2024-04-24T01:55:32.408 INFO:tasks.rados.rados.0.smithi079.stdout:7: finishing write tid 4 to smithi07929762-7 2024-04-24T01:55:32.408 INFO:tasks.rados.rados.0.smithi079.stdout:7: oid 7 updating version 0 to 3 2024-04-24T01:55:32.408 INFO:tasks.rados.rados.0.smithi079.stdout:7: oid 7 version 3 is already newer than 1 2024-04-24T01:55:32.408 INFO:tasks.rados.rados.0.smithi079.stdout:7: oid 7 version 3 is already newer than 3 2024-04-24T01:55:32.409 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 7 v 3 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-04-24T01:55:32.409 INFO:tasks.rados.rados.0.smithi079.stdout:7: left oid 7 (ObjNum 6 snap 0 seq_num 6) 2024-04-24T01:55:32.409 INFO:tasks.rados.rados.0.smithi079.stdout:8: finishing write tid 1 to smithi07929762-8 2024-04-24T01:55:32.409 INFO:tasks.rados.rados.0.smithi079.stdout:2: finishing write tid 3 to smithi07929762-2 2024-04-24T01:55:32.409 INFO:tasks.rados.rados.0.smithi079.stdout:5: finishing write tid 2 to smithi07929762-5 2024-04-24T01:55:32.409 INFO:tasks.rados.rados.0.smithi079.stdout:1: done (15 left) 2024-04-24T01:55:32.409 INFO:tasks.rados.rados.0.smithi079.stdout:6: done (14 left) 2024-04-24T01:55:32.409 INFO:tasks.rados.rados.0.smithi079.stdout:7: done (13 left) 2024-04-24T01:55:32.409 INFO:tasks.rados.rados.0.smithi079.stdout:17: write initial oid 17 2024-04-24T01:55:32.409 INFO:tasks.rados.rados.0.smithi079.stdout:17: seq_num 16 ranges {668912=636690,1903006=365906} 2024-04-24T01:55:32.416 INFO:tasks.rados.rados.0.smithi079.stdout:17: writing smithi07929762-17 from 668912 to 1305602 tid 1 2024-04-24T01:55:32.421 INFO:tasks.rados.rados.0.smithi079.stdout:17: writing smithi07929762-17 from 1903006 to 2268912 tid 2 2024-04-24T01:55:32.421 INFO:tasks.rados.rados.0.smithi079.stdout:18: write initial oid 18 2024-04-24T01:55:32.422 INFO:tasks.rados.rados.0.smithi079.stdout:18: seq_num 17 ranges {685719=711939,1845135=440584} 2024-04-24T01:55:32.429 INFO:tasks.rados.rados.0.smithi079.stdout:18: writing smithi07929762-18 from 685719 to 1397658 tid 1 2024-04-24T01:55:32.435 INFO:tasks.rados.rados.0.smithi079.stdout:18: writing smithi07929762-18 from 1845135 to 2285719 tid 2 2024-04-24T01:55:32.435 INFO:tasks.rados.rados.0.smithi079.stdout:19: write initial oid 19 2024-04-24T01:55:32.435 INFO:tasks.rados.rados.0.smithi079.stdout:19: seq_num 18 ranges {702526=787188,1903617=398909} 2024-04-24T01:55:32.443 INFO:tasks.rados.rados.0.smithi079.stdout:19: writing smithi07929762-19 from 702526 to 1489714 tid 1 2024-04-24T01:55:32.450 INFO:tasks.rados.rados.0.smithi079.stdout:19: writing smithi07929762-19 from 1903617 to 2302526 tid 2 2024-04-24T01:55:32.450 INFO:tasks.rados.rados.0.smithi079.stdout: waiting on 16 2024-04-24T01:55:32.450 INFO:tasks.rados.rados.0.smithi079.stdout:2: finishing write tid 4 to smithi07929762-2 2024-04-24T01:55:32.450 INFO:tasks.rados.rados.0.smithi079.stdout:2: oid 2 updating version 0 to 1 2024-04-24T01:55:32.450 INFO:tasks.rados.rados.0.smithi079.stdout:2: oid 2 updating version 1 to 3 2024-04-24T01:55:32.450 INFO:tasks.rados.rados.0.smithi079.stdout:2: oid 2 version 3 is already newer than 3 2024-04-24T01:55:32.450 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 2 v 3 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-04-24T01:55:32.450 INFO:tasks.rados.rados.0.smithi079.stdout:2: left oid 2 (ObjNum 1 snap 0 seq_num 1) 2024-04-24T01:55:32.450 INFO:tasks.rados.rados.0.smithi079.stdout:5: finishing write tid 3 to smithi07929762-5 2024-04-24T01:55:32.450 INFO:tasks.rados.rados.0.smithi079.stdout:2: done (15 left) 2024-04-24T01:55:32.450 INFO:tasks.rados.rados.0.smithi079.stdout:20: write initial oid 20 2024-04-24T01:55:32.451 INFO:tasks.rados.rados.0.smithi079.stdout:20: seq_num 19 ranges {719333=462437,1962099=357234} 2024-04-24T01:55:32.458 INFO:tasks.rados.rados.0.smithi079.stdout:20: writing smithi07929762-20 from 719333 to 1181770 tid 1 2024-04-24T01:55:32.466 INFO:tasks.rados.rados.0.smithi079.stdout:20: writing smithi07929762-20 from 1962099 to 2319333 tid 2 2024-04-24T01:55:32.466 INFO:tasks.rados.rados.0.smithi079.stdout: waiting on 16 2024-04-24T01:55:32.466 INFO:tasks.rados.rados.0.smithi079.stdout:4: finishing write tid 2 to smithi07929762-4 2024-04-24T01:55:32.467 INFO:tasks.rados.rados.0.smithi079.stdout:5: finishing write tid 4 to smithi07929762-5 2024-04-24T01:55:32.467 INFO:tasks.rados.rados.0.smithi079.stdout:5: oid 5 updating version 0 to 1 2024-04-24T01:55:32.467 INFO:tasks.rados.rados.0.smithi079.stdout:5: oid 5 updating version 1 to 3 2024-04-24T01:55:32.467 INFO:tasks.rados.rados.0.smithi079.stdout:5: oid 5 version 3 is already newer than 3 2024-04-24T01:55:32.467 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 5 v 3 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-04-24T01:55:32.467 INFO:tasks.rados.rados.0.smithi079.stdout:5: left oid 5 (ObjNum 4 snap 0 seq_num 4) 2024-04-24T01:55:32.467 INFO:tasks.rados.rados.0.smithi079.stdout:4: finishing write tid 3 to smithi07929762-4 2024-04-24T01:55:32.467 INFO:tasks.rados.rados.0.smithi079.stdout:4: finishing write tid 4 to smithi07929762-4 2024-04-24T01:55:32.467 INFO:tasks.rados.rados.0.smithi079.stdout:4: oid 4 updating version 0 to 3 2024-04-24T01:55:32.467 INFO:tasks.rados.rados.0.smithi079.stdout:4: oid 4 version 3 is already newer than 1 2024-04-24T01:55:32.467 INFO:tasks.rados.rados.0.smithi079.stdout:4: oid 4 version 3 is already newer than 3 2024-04-24T01:55:32.467 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 4 v 3 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-04-24T01:55:32.467 INFO:tasks.rados.rados.0.smithi079.stdout:4: left oid 4 (ObjNum 3 snap 0 seq_num 3) 2024-04-24T01:55:32.467 INFO:tasks.rados.rados.0.smithi079.stdout:12: finishing write tid 1 to smithi07929762-12 2024-04-24T01:55:32.467 INFO:tasks.rados.rados.0.smithi079.stdout:10: finishing write tid 1 to smithi07929762-10 2024-04-24T01:55:32.467 INFO:tasks.rados.rados.0.smithi079.stdout:11: finishing write tid 1 to smithi07929762-11 2024-04-24T01:55:32.468 INFO:tasks.rados.rados.0.smithi079.stdout:9: finishing write tid 1 to smithi07929762-9 2024-04-24T01:55:32.468 INFO:tasks.rados.rados.0.smithi079.stdout:3: finishing write tid 1 to smithi07929762-3 2024-04-24T01:55:32.468 INFO:tasks.rados.rados.0.smithi079.stdout:13: finishing write tid 1 to smithi07929762-13 2024-04-24T01:55:32.468 INFO:tasks.rados.rados.0.smithi079.stdout:8: finishing write tid 2 to smithi07929762-8 2024-04-24T01:55:32.468 INFO:tasks.rados.rados.0.smithi079.stdout:8: finishing write tid 3 to smithi07929762-8 2024-04-24T01:55:32.468 INFO:tasks.rados.rados.0.smithi079.stdout:8: finishing write tid 4 to smithi07929762-8 2024-04-24T01:55:32.468 INFO:tasks.rados.rados.0.smithi079.stdout:8: oid 8 updating version 0 to 1 2024-04-24T01:55:32.468 INFO:tasks.rados.rados.0.smithi079.stdout:8: oid 8 updating version 1 to 3 2024-04-24T01:55:32.468 INFO:tasks.rados.rados.0.smithi079.stdout:8: oid 8 version 3 is already newer than 3 2024-04-24T01:55:32.468 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 8 v 3 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-04-24T01:55:32.468 INFO:tasks.rados.rados.0.smithi079.stdout:8: left oid 8 (ObjNum 7 snap 0 seq_num 7) 2024-04-24T01:55:32.468 INFO:tasks.rados.rados.0.smithi079.stdout:4: done (15 left) 2024-04-24T01:55:32.468 INFO:tasks.rados.rados.0.smithi079.stdout:5: done (14 left) 2024-04-24T01:55:32.468 INFO:tasks.rados.rados.0.smithi079.stdout:8: done (13 left) 2024-04-24T01:55:32.468 INFO:tasks.rados.rados.0.smithi079.stdout:21: write initial oid 21 2024-04-24T01:55:32.469 INFO:tasks.rados.rados.0.smithi079.stdout:21: seq_num 20 ranges {736140=537686,2020581=315559} 2024-04-24T01:55:32.473 INFO:tasks.rados.rados.0.smithi079.stdout:21: writing smithi07929762-21 from 736140 to 1273826 tid 1 2024-04-24T01:55:32.480 INFO:tasks.rados.rados.0.smithi079.stdout:21: writing smithi07929762-21 from 2020581 to 2336140 tid 2 2024-04-24T01:55:32.480 INFO:tasks.rados.rados.0.smithi079.stdout:22: write initial oid 22 2024-04-24T01:55:32.480 INFO:tasks.rados.rados.0.smithi079.stdout:22: seq_num 21 ranges {752947=612935,1962710=390237} 2024-04-24T01:55:32.489 INFO:tasks.rados.rados.0.smithi079.stdout:22: writing smithi07929762-22 from 752947 to 1365882 tid 1 2024-04-24T01:55:32.495 INFO:tasks.rados.rados.0.smithi079.stdout:22: writing smithi07929762-22 from 1962710 to 2352947 tid 2 2024-04-24T01:55:32.495 INFO:tasks.rados.rados.0.smithi079.stdout:23: write initial oid 23 2024-04-24T01:55:32.495 INFO:tasks.rados.rados.0.smithi079.stdout:23: seq_num 22 ranges {769754=688184,2021192=348562} 2024-04-24T01:55:32.507 INFO:tasks.rados.rados.0.smithi079.stdout:23: writing smithi07929762-23 from 769754 to 1457938 tid 1 2024-04-24T01:55:32.508 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T01:55:32.511 INFO:tasks.rados.rados.0.smithi079.stdout:23: writing smithi07929762-23 from 2021192 to 2369754 tid 2 2024-04-24T01:55:32.511 INFO:tasks.rados.rados.0.smithi079.stdout: waiting on 16 2024-04-24T01:55:32.511 INFO:tasks.rados.rados.0.smithi079.stdout:10: finishing write tid 2 to smithi07929762-10 2024-04-24T01:55:32.511 INFO:tasks.rados.rados.0.smithi079.stdout:3: finishing write tid 2 to smithi07929762-3 2024-04-24T01:55:32.511 INFO:tasks.rados.rados.0.smithi079.stdout:3: finishing write tid 3 to smithi07929762-3 2024-04-24T01:55:32.511 INFO:tasks.rados.rados.0.smithi079.stdout:12: finishing write tid 2 to smithi07929762-12 2024-04-24T01:55:32.511 INFO:tasks.rados.rados.0.smithi079.stdout:11: finishing write tid 2 to smithi07929762-11 2024-04-24T01:55:32.511 INFO:tasks.rados.rados.0.smithi079.stdout:10: finishing write tid 3 to smithi07929762-10 2024-04-24T01:55:32.511 INFO:tasks.rados.rados.0.smithi079.stdout:10: finishing write tid 4 to smithi07929762-10 2024-04-24T01:55:32.511 INFO:tasks.rados.rados.0.smithi079.stdout:10: oid 10 updating version 0 to 1 2024-04-24T01:55:32.511 INFO:tasks.rados.rados.0.smithi079.stdout:10: oid 10 updating version 1 to 3 2024-04-24T01:55:32.511 INFO:tasks.rados.rados.0.smithi079.stdout:10: oid 10 version 3 is already newer than 3 2024-04-24T01:55:32.512 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 10 v 3 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-04-24T01:55:32.512 INFO:tasks.rados.rados.0.smithi079.stdout:10: left oid 10 (ObjNum 9 snap 0 seq_num 9) 2024-04-24T01:55:32.512 INFO:tasks.rados.rados.0.smithi079.stdout:3: finishing write tid 4 to smithi07929762-3 2024-04-24T01:55:32.512 INFO:tasks.rados.rados.0.smithi079.stdout:3: oid 3 updating version 0 to 6 2024-04-24T01:55:32.512 INFO:tasks.rados.rados.0.smithi079.stdout:3: oid 3 version 6 is already newer than 4 2024-04-24T01:55:32.512 INFO:tasks.rados.rados.0.smithi079.stdout:3: oid 3 version 6 is already newer than 6 2024-04-24T01:55:32.512 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 3 v 6 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-04-24T01:55:32.512 INFO:tasks.rados.rados.0.smithi079.stdout:3: left oid 3 (ObjNum 2 snap 0 seq_num 2) 2024-04-24T01:55:32.512 INFO:tasks.rados.rados.0.smithi079.stdout:9: finishing write tid 2 to smithi07929762-9 2024-04-24T01:55:32.512 INFO:tasks.rados.rados.0.smithi079.stdout:3: done (15 left) 2024-04-24T01:55:32.512 INFO:tasks.rados.rados.0.smithi079.stdout:10: done (14 left) 2024-04-24T01:55:32.512 INFO:tasks.rados.rados.0.smithi079.stdout:24: write initial oid 24 2024-04-24T01:55:32.512 INFO:tasks.rados.rados.0.smithi079.stdout:24: seq_num 23 ranges {786561=479786,1796027=467664} 2024-04-24T01:55:32.519 INFO:tasks.rados.rados.0.smithi079.stdout:24: writing smithi07929762-24 from 786561 to 1266347 tid 1 2024-04-24T01:55:32.527 INFO:tasks.rados.rados.0.smithi079.stdout:24: writing smithi07929762-24 from 1796027 to 2263691 tid 2 2024-04-24T01:55:32.527 INFO:tasks.rados.rados.0.smithi079.stdout:25: write initial oid 25 2024-04-24T01:55:32.527 INFO:tasks.rados.rados.0.smithi079.stdout:25: seq_num 24 ranges {403368=555035,1454509=727675} 2024-04-24T01:55:32.532 INFO:tasks.rados.rados.0.smithi079.stdout:25: writing smithi07929762-25 from 403368 to 958403 tid 1 2024-04-24T01:55:32.540 INFO:tasks.rados.rados.0.smithi079.stdout:25: writing smithi07929762-25 from 1454509 to 2182184 tid 2 2024-04-24T01:55:32.540 INFO:tasks.rados.rados.0.smithi079.stdout: waiting on 16 2024-04-24T01:55:32.540 INFO:tasks.rados.rados.0.smithi079.stdout:9: finishing write tid 3 to smithi07929762-9 2024-04-24T01:55:32.540 INFO:tasks.rados.rados.0.smithi079.stdout:11: finishing write tid 3 to smithi07929762-11 2024-04-24T01:55:32.540 INFO:tasks.rados.rados.0.smithi079.stdout:9: finishing write tid 4 to smithi07929762-9 2024-04-24T01:55:32.541 INFO:tasks.rados.rados.0.smithi079.stdout:9: oid 9 updating version 0 to 3 2024-04-24T01:55:32.541 INFO:tasks.rados.rados.0.smithi079.stdout:9: oid 9 version 3 is already newer than 1 2024-04-24T01:55:32.541 INFO:tasks.rados.rados.0.smithi079.stdout:9: oid 9 version 3 is already newer than 3 2024-04-24T01:55:32.541 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 9 v 3 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-04-24T01:55:32.541 INFO:tasks.rados.rados.0.smithi079.stdout:9: left oid 9 (ObjNum 8 snap 0 seq_num 8) 2024-04-24T01:55:32.541 INFO:tasks.rados.rados.0.smithi079.stdout:13: finishing write tid 2 to smithi07929762-13 2024-04-24T01:55:32.541 INFO:tasks.rados.rados.0.smithi079.stdout:13: finishing write tid 3 to smithi07929762-13 2024-04-24T01:55:32.541 INFO:tasks.rados.rados.0.smithi079.stdout:12: finishing write tid 3 to smithi07929762-12 2024-04-24T01:55:32.541 INFO:tasks.rados.rados.0.smithi079.stdout:9: done (15 left) 2024-04-24T01:55:32.541 INFO:tasks.rados.rados.0.smithi079.stdout:26: write initial oid 26 2024-04-24T01:55:32.541 INFO:tasks.rados.rados.0.smithi079.stdout:26: seq_num 25 ranges {420175=630284,1796638=471333} 2024-04-24T01:55:32.548 INFO:tasks.rados.rados.0.smithi079.stdout:26: writing smithi07929762-26 from 420175 to 1050459 tid 1 2024-04-24T01:55:32.558 INFO:tasks.rados.rados.0.smithi079.stdout:26: writing smithi07929762-26 from 1796638 to 2267971 tid 2 2024-04-24T01:55:32.558 INFO:tasks.rados.rados.0.smithi079.stdout: waiting on 16 2024-04-24T01:55:32.558 INFO:tasks.rados.rados.0.smithi079.stdout:13: finishing write tid 4 to smithi07929762-13 2024-04-24T01:55:32.558 INFO:tasks.rados.rados.0.smithi079.stdout:13: oid 13 updating version 0 to 3 2024-04-24T01:55:32.558 INFO:tasks.rados.rados.0.smithi079.stdout:13: oid 13 version 3 is already newer than 3 2024-04-24T01:55:32.558 INFO:tasks.rados.rados.0.smithi079.stdout:13: oid 13 version 3 is already newer than 1 2024-04-24T01:55:32.558 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 13 v 3 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-04-24T01:55:32.558 INFO:tasks.rados.rados.0.smithi079.stdout:13: left oid 13 (ObjNum 12 snap 0 seq_num 12) 2024-04-24T01:55:32.558 INFO:tasks.rados.rados.0.smithi079.stdout:12: finishing write tid 4 to smithi07929762-12 2024-04-24T01:55:32.559 INFO:tasks.rados.rados.0.smithi079.stdout:12: oid 12 updating version 0 to 6 2024-04-24T01:55:32.559 INFO:tasks.rados.rados.0.smithi079.stdout:12: oid 12 version 6 is already newer than 6 2024-04-24T01:55:32.559 INFO:tasks.rados.rados.0.smithi079.stdout:12: oid 12 version 6 is already newer than 4 2024-04-24T01:55:32.559 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 12 v 6 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-04-24T01:55:32.559 INFO:tasks.rados.rados.0.smithi079.stdout:12: left oid 12 (ObjNum 11 snap 0 seq_num 11) 2024-04-24T01:55:32.559 INFO:tasks.rados.rados.0.smithi079.stdout:11: finishing write tid 4 to smithi07929762-11 2024-04-24T01:55:32.559 INFO:tasks.rados.rados.0.smithi079.stdout:11: oid 11 updating version 0 to 3 2024-04-24T01:55:32.559 INFO:tasks.rados.rados.0.smithi079.stdout:11: oid 11 version 3 is already newer than 3 2024-04-24T01:55:32.559 INFO:tasks.rados.rados.0.smithi079.stdout:11: oid 11 version 3 is already newer than 1 2024-04-24T01:55:32.559 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 11 v 3 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-04-24T01:55:32.559 INFO:tasks.rados.rados.0.smithi079.stdout:11: left oid 11 (ObjNum 10 snap 0 seq_num 10) 2024-04-24T01:55:32.559 INFO:tasks.rados.rados.0.smithi079.stdout:16: finishing write tid 1 to smithi07929762-16 2024-04-24T01:55:32.559 INFO:tasks.rados.rados.0.smithi079.stdout:11: done (15 left) 2024-04-24T01:55:32.559 INFO:tasks.rados.rados.0.smithi079.stdout:12: done (14 left) 2024-04-24T01:55:32.559 INFO:tasks.rados.rados.0.smithi079.stdout:13: done (13 left) 2024-04-24T01:55:32.560 INFO:tasks.rados.rados.0.smithi079.stdout:27: write initial oid 27 2024-04-24T01:55:32.560 INFO:tasks.rados.rados.0.smithi079.stdout:27: seq_num 26 ranges {436982=705533,1855120=581862} 2024-04-24T01:55:32.564 INFO:tasks.rados.rados.0.smithi079.stdout:27: writing smithi07929762-27 from 436982 to 1142515 tid 1 2024-04-24T01:55:32.573 INFO:tasks.rados.rados.0.smithi079.stdout:27: writing smithi07929762-27 from 1855120 to 2436982 tid 2 2024-04-24T01:55:32.574 INFO:tasks.rados.rados.0.smithi079.stdout:28: write initial oid 28 2024-04-24T01:55:32.574 INFO:tasks.rados.rados.0.smithi079.stdout:28: seq_num 27 ranges {453789=780782,1913602=475002} 2024-04-24T01:55:32.581 INFO:tasks.rados.rados.0.smithi079.stdout:28: writing smithi07929762-28 from 453789 to 1234571 tid 1 2024-04-24T01:55:32.588 INFO:tasks.rados.rados.0.smithi079.stdout:28: writing smithi07929762-28 from 1913602 to 2388604 tid 2 2024-04-24T01:55:32.588 INFO:tasks.rados.rados.0.smithi079.stdout:29: write initial oid 29 2024-04-24T01:55:32.589 INFO:tasks.rados.rados.0.smithi079.stdout:29: seq_num 28 ranges {470596=456031,1572084=618660} 2024-04-24T01:55:32.594 INFO:tasks.rados.rados.0.smithi079.stdout:29: writing smithi07929762-29 from 470596 to 926627 tid 1 2024-04-24T01:55:32.604 INFO:tasks.rados.rados.0.smithi079.stdout:29: writing smithi07929762-29 from 1572084 to 2190744 tid 2 2024-04-24T01:55:32.604 INFO:tasks.rados.rados.0.smithi079.stdout: waiting on 16 2024-04-24T01:55:32.604 INFO:tasks.rados.rados.0.smithi079.stdout:15: finishing write tid 1 to smithi07929762-15 2024-04-24T01:55:32.604 INFO:tasks.rados.rados.0.smithi079.stdout:23: finishing write tid 1 to smithi07929762-23 2024-04-24T01:55:32.604 INFO:tasks.rados.rados.0.smithi079.stdout:21: finishing write tid 1 to smithi07929762-21 2024-04-24T01:55:32.604 INFO:tasks.rados.rados.0.smithi079.stdout:20: finishing write tid 1 to smithi07929762-20 2024-04-24T01:55:32.605 INFO:tasks.rados.rados.0.smithi079.stdout:14: finishing write tid 1 to smithi07929762-14 2024-04-24T01:55:32.605 INFO:tasks.rados.rados.0.smithi079.stdout:15: finishing write tid 2 to smithi07929762-15 2024-04-24T01:55:32.632 INFO:tasks.rados.rados.0.smithi079.stdout:16: finishing write tid 2 to smithi07929762-16 2024-04-24T01:55:32.632 INFO:tasks.rados.rados.0.smithi079.stdout:15: finishing write tid 3 to smithi07929762-15 2024-04-24T01:55:32.636 INFO:tasks.rados.rados.0.smithi079.stdout:15: finishing write tid 4 to smithi07929762-15 2024-04-24T01:55:32.636 INFO:tasks.rados.rados.0.smithi079.stdout:15: oid 15 updating version 0 to 7 2024-04-24T01:55:32.636 INFO:tasks.rados.rados.0.smithi079.stdout:15: oid 15 updating version 7 to 9 2024-04-24T01:55:32.636 INFO:tasks.rados.rados.0.smithi079.stdout:15: oid 15 version 9 is already newer than 9 2024-04-24T01:55:32.637 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 15 v 9 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-04-24T01:55:32.637 INFO:tasks.rados.rados.0.smithi079.stdout:15: left oid 15 (ObjNum 14 snap 0 seq_num 14) 2024-04-24T01:55:32.637 INFO:tasks.rados.rados.0.smithi079.stdout:15: done (15 left) 2024-04-24T01:55:32.637 INFO:tasks.rados.rados.0.smithi079.stdout:30: write initial oid 30 2024-04-24T01:55:32.637 INFO:tasks.rados.rados.0.smithi079.stdout:30: seq_num 29 ranges {487403=531280,1514213=478671} 2024-04-24T01:55:32.644 INFO:tasks.rados.rados.0.smithi079.stdout:30: writing smithi07929762-30 from 487403 to 1018683 tid 1 2024-04-24T01:55:32.652 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:32.650+0000 7f44cf19d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:32.653 INFO:tasks.rados.rados.0.smithi079.stdout:30: writing smithi07929762-30 from 1514213 to 1992884 tid 2 2024-04-24T01:55:32.653 INFO:tasks.rados.rados.0.smithi079.stdout: waiting on 16 2024-04-24T01:55:32.653 INFO:tasks.rados.rados.0.smithi079.stdout:21: finishing write tid 2 to smithi07929762-21 2024-04-24T01:55:32.653 INFO:tasks.rados.rados.0.smithi079.stdout:16: finishing write tid 3 to smithi07929762-16 2024-04-24T01:55:32.656 INFO:tasks.rados.rados.0.smithi079.stdout:16: finishing write tid 4 to smithi07929762-16 2024-04-24T01:55:32.656 INFO:tasks.rados.rados.0.smithi079.stdout:16: oid 16 updating version 0 to 9 2024-04-24T01:55:32.656 INFO:tasks.rados.rados.0.smithi079.stdout:16: oid 16 version 9 is already newer than 9 2024-04-24T01:55:32.656 INFO:tasks.rados.rados.0.smithi079.stdout:16: oid 16 version 9 is already newer than 7 2024-04-24T01:55:32.656 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 16 v 9 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-04-24T01:55:32.656 INFO:tasks.rados.rados.0.smithi079.stdout:16: left oid 16 (ObjNum 15 snap 0 seq_num 15) 2024-04-24T01:55:32.656 INFO:tasks.rados.rados.0.smithi079.stdout:16: done (15 left) 2024-04-24T01:55:32.656 INFO:tasks.rados.rados.0.smithi079.stdout:31: write initial oid 31 2024-04-24T01:55:32.656 INFO:tasks.rados.rados.0.smithi079.stdout:31: seq_num 30 ranges {504210=606529,1572695=622329} 2024-04-24T01:55:32.659 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:32.659+0000 7f44cf19d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:32.664 INFO:tasks.rados.rados.0.smithi079.stdout:31: writing smithi07929762-31 from 504210 to 1110739 tid 1 2024-04-24T01:55:32.674 INFO:tasks.rados.rados.0.smithi079.stdout:31: writing smithi07929762-31 from 1572695 to 2195024 tid 2 2024-04-24T01:55:32.674 INFO:tasks.rados.rados.0.smithi079.stdout: waiting on 16 2024-04-24T01:55:32.674 INFO:tasks.rados.rados.0.smithi079.stdout:20: finishing write tid 2 to smithi07929762-20 2024-04-24T01:55:32.674 INFO:tasks.rados.rados.0.smithi079.stdout:21: finishing write tid 3 to smithi07929762-21 2024-04-24T01:55:32.674 INFO:tasks.rados.rados.0.smithi079.stdout:22: finishing write tid 1 to smithi07929762-22 2024-04-24T01:55:32.674 INFO:tasks.rados.rados.0.smithi079.stdout:27: finishing write tid 1 to smithi07929762-27 2024-04-24T01:55:32.674 INFO:tasks.rados.rados.0.smithi079.stdout:19: finishing write tid 1 to smithi07929762-19 2024-04-24T01:55:32.674 INFO:tasks.rados.rados.0.smithi079.stdout:21: finishing write tid 4 to smithi07929762-21 2024-04-24T01:55:32.674 INFO:tasks.rados.rados.0.smithi079.stdout:21: oid 21 updating version 0 to 6 2024-04-24T01:55:32.674 INFO:tasks.rados.rados.0.smithi079.stdout:21: oid 21 version 6 is already newer than 6 2024-04-24T01:55:32.674 INFO:tasks.rados.rados.0.smithi079.stdout:21: oid 21 version 6 is already newer than 4 2024-04-24T01:55:32.674 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 21 v 6 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-04-24T01:55:32.674 INFO:tasks.rados.rados.0.smithi079.stdout:21: left oid 21 (ObjNum 20 snap 0 seq_num 20) 2024-04-24T01:55:32.674 INFO:tasks.rados.rados.0.smithi079.stdout:25: finishing write tid 1 to smithi07929762-25 2024-04-24T01:55:32.674 INFO:tasks.rados.rados.0.smithi079.stdout:21: done (15 left) 2024-04-24T01:55:32.675 INFO:tasks.rados.rados.0.smithi079.stdout:32: write initial oid 32 2024-04-24T01:55:32.675 INFO:tasks.rados.rados.0.smithi079.stdout:32: seq_num 31 ranges {521017=798131,1747530=482340} 2024-04-24T01:55:32.685 INFO:tasks.rados.rados.0.smithi079.stdout:32: writing smithi07929762-32 from 521017 to 1319148 tid 1 2024-04-24T01:55:32.694 INFO:tasks.rados.rados.0.smithi079.stdout:32: writing smithi07929762-32 from 1747530 to 2229870 tid 2 2024-04-24T01:55:32.694 INFO:tasks.rados.rados.0.smithi079.stdout: waiting on 16 2024-04-24T01:55:32.694 INFO:tasks.rados.rados.0.smithi079.stdout:23: finishing write tid 2 to smithi07929762-23 2024-04-24T01:55:32.694 INFO:tasks.rados.rados.0.smithi079.stdout:14: finishing write tid 2 to smithi07929762-14 2024-04-24T01:55:32.694 INFO:tasks.rados.rados.0.smithi079.stdout:20: finishing write tid 3 to smithi07929762-20 2024-04-24T01:55:32.694 INFO:tasks.rados.rados.0.smithi079.stdout:23: finishing write tid 3 to smithi07929762-23 2024-04-24T01:55:32.695 INFO:tasks.rados.rados.0.smithi079.stdout:23: finishing write tid 4 to smithi07929762-23 2024-04-24T01:55:32.695 INFO:tasks.rados.rados.0.smithi079.stdout:23: oid 23 updating version 0 to 6 2024-04-24T01:55:32.695 INFO:tasks.rados.rados.0.smithi079.stdout:23: oid 23 version 6 is already newer than 4 2024-04-24T01:55:32.695 INFO:tasks.rados.rados.0.smithi079.stdout:23: oid 23 version 6 is already newer than 6 2024-04-24T01:55:32.695 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 23 v 6 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-04-24T01:55:32.695 INFO:tasks.rados.rados.0.smithi079.stdout:23: left oid 23 (ObjNum 22 snap 0 seq_num 22) 2024-04-24T01:55:32.695 INFO:tasks.rados.rados.0.smithi079.stdout:23: done (15 left) 2024-04-24T01:55:32.695 INFO:tasks.rados.rados.0.smithi079.stdout:33: write initial oid 33 2024-04-24T01:55:32.695 INFO:tasks.rados.rados.0.smithi079.stdout:33: seq_num 32 ranges {537824=473380,1806012=625998} 2024-04-24T01:55:32.701 INFO:tasks.rados.rados.0.smithi079.stdout:33: writing smithi07929762-33 from 537824 to 1011204 tid 1 2024-04-24T01:55:32.713 INFO:tasks.rados.rados.0.smithi079.stdout:33: writing smithi07929762-33 from 1806012 to 2432010 tid 2 2024-04-24T01:55:32.713 INFO:tasks.rados.rados.0.smithi079.stdout: waiting on 16 2024-04-24T01:55:32.713 INFO:tasks.rados.rados.0.smithi079.stdout:17: finishing write tid 1 to smithi07929762-17 2024-04-24T01:55:32.713 INFO:tasks.rados.rados.0.smithi079.stdout:20: finishing write tid 4 to smithi07929762-20 2024-04-24T01:55:32.713 INFO:tasks.rados.rados.0.smithi079.stdout:20: oid 20 updating version 0 to 6 2024-04-24T01:55:32.713 INFO:tasks.rados.rados.0.smithi079.stdout:20: oid 20 version 6 is already newer than 4 2024-04-24T01:55:32.713 INFO:tasks.rados.rados.0.smithi079.stdout:20: oid 20 version 6 is already newer than 6 2024-04-24T01:55:32.713 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 20 v 6 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-04-24T01:55:32.713 INFO:tasks.rados.rados.0.smithi079.stdout:20: left oid 20 (ObjNum 19 snap 0 seq_num 19) 2024-04-24T01:55:32.713 INFO:tasks.rados.rados.0.smithi079.stdout:14: finishing write tid 3 to smithi07929762-14 2024-04-24T01:55:32.713 INFO:tasks.rados.rados.0.smithi079.stdout:20: done (15 left) 2024-04-24T01:55:32.713 INFO:tasks.rados.rados.0.smithi079.stdout:34: write initial oid 34 2024-04-24T01:55:32.713 INFO:tasks.rados.rados.0.smithi079.stdout:34: seq_num 33 ranges {554631=548629,1748141=486009} 2024-04-24T01:55:32.721 INFO:tasks.rados.rados.0.smithi079.stdout:34: writing smithi07929762-34 from 554631 to 1103260 tid 1 2024-04-24T01:55:32.730 INFO:tasks.rados.rados.0.smithi079.stdout:34: writing smithi07929762-34 from 1748141 to 2234150 tid 2 2024-04-24T01:55:32.730 INFO:tasks.rados.rados.0.smithi079.stdout: waiting on 16 2024-04-24T01:55:32.731 INFO:tasks.rados.rados.0.smithi079.stdout:27: finishing write tid 2 to smithi07929762-27 2024-04-24T01:55:32.731 INFO:tasks.rados.rados.0.smithi079.stdout:14: finishing write tid 4 to smithi07929762-14 2024-04-24T01:55:32.731 INFO:tasks.rados.rados.0.smithi079.stdout:14: oid 14 updating version 0 to 6 2024-04-24T01:55:32.731 INFO:tasks.rados.rados.0.smithi079.stdout:14: oid 14 version 6 is already newer than 6 2024-04-24T01:55:32.731 INFO:tasks.rados.rados.0.smithi079.stdout:14: oid 14 version 6 is already newer than 4 2024-04-24T01:55:32.731 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 14 v 6 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-04-24T01:55:32.731 INFO:tasks.rados.rados.0.smithi079.stdout:14: left oid 14 (ObjNum 13 snap 0 seq_num 13) 2024-04-24T01:55:32.731 INFO:tasks.rados.rados.0.smithi079.stdout:14: done (15 left) 2024-04-24T01:55:32.731 INFO:tasks.rados.rados.0.smithi079.stdout:35: write initial oid 35 2024-04-24T01:55:32.731 INFO:tasks.rados.rados.0.smithi079.stdout:35: seq_num 34 ranges {571438=623878,1806623=629667} 2024-04-24T01:55:32.737 INFO:tasks.rados.rados.0.smithi079.stdout:35: writing smithi07929762-35 from 571438 to 1195316 tid 1 2024-04-24T01:55:32.746 INFO:tasks.rados.rados.0.smithi079.stdout:35: writing smithi07929762-35 from 1806623 to 2436290 tid 2 2024-04-24T01:55:32.746 INFO:tasks.rados.rados.0.smithi079.stdout: waiting on 16 2024-04-24T01:55:32.746 INFO:tasks.rados.rados.0.smithi079.stdout:22: finishing write tid 2 to smithi07929762-22 2024-04-24T01:55:32.747 INFO:tasks.rados.rados.0.smithi079.stdout:27: finishing write tid 3 to smithi07929762-27 2024-04-24T01:55:32.747 INFO:tasks.rados.rados.0.smithi079.stdout:27: finishing write tid 4 to smithi07929762-27 2024-04-24T01:55:32.747 INFO:tasks.rados.rados.0.smithi079.stdout:27: oid 27 updating version 0 to 1 2024-04-24T01:55:32.747 INFO:tasks.rados.rados.0.smithi079.stdout:27: oid 27 updating version 1 to 3 2024-04-24T01:55:32.747 INFO:tasks.rados.rados.0.smithi079.stdout:27: oid 27 version 3 is already newer than 3 2024-04-24T01:55:32.747 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 27 v 3 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-04-24T01:55:32.747 INFO:tasks.rados.rados.0.smithi079.stdout:27: left oid 27 (ObjNum 26 snap 0 seq_num 26) 2024-04-24T01:55:32.747 INFO:tasks.rados.rados.0.smithi079.stdout:28: finishing write tid 1 to smithi07929762-28 2024-04-24T01:55:32.748 INFO:tasks.rados.rados.0.smithi079.stdout:27: done (15 left) 2024-04-24T01:55:32.748 INFO:tasks.rados.rados.0.smithi079.stdout:36: write initial oid 36 2024-04-24T01:55:32.748 INFO:tasks.rados.rados.0.smithi079.stdout:36: seq_num 35 ranges {588245=699127,1865105=489678} 2024-04-24T01:55:32.752 INFO:tasks.rados.rados.0.smithi079.stdout:36: writing smithi07929762-36 from 588245 to 1287372 tid 1 2024-04-24T01:55:32.761 INFO:tasks.rados.rados.0.smithi079.stdout:36: writing smithi07929762-36 from 1865105 to 2354783 tid 2 2024-04-24T01:55:32.762 INFO:tasks.rados.rados.0.smithi079.stdout: waiting on 16 2024-04-24T01:55:32.762 INFO:tasks.rados.rados.0.smithi079.stdout:18: finishing write tid 1 to smithi07929762-18 2024-04-24T01:55:32.762 INFO:tasks.rados.rados.0.smithi079.stdout:17: finishing write tid 2 to smithi07929762-17 2024-04-24T01:55:32.771 INFO:tasks.rados.rados.0.smithi079.stdout:17: finishing write tid 3 to smithi07929762-17 2024-04-24T01:55:32.771 INFO:tasks.rados.rados.0.smithi079.stdout:25: finishing write tid 2 to smithi07929762-25 2024-04-24T01:55:32.773 INFO:tasks.rados.rados.0.smithi079.stdout:17: finishing write tid 4 to smithi07929762-17 2024-04-24T01:55:32.773 INFO:tasks.rados.rados.0.smithi079.stdout:17: oid 17 updating version 0 to 12 2024-04-24T01:55:32.774 INFO:tasks.rados.rados.0.smithi079.stdout:17: oid 17 version 12 is already newer than 10 2024-04-24T01:55:32.774 INFO:tasks.rados.rados.0.smithi079.stdout:17: oid 17 version 12 is already newer than 12 2024-04-24T01:55:32.774 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 17 v 12 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-04-24T01:55:32.774 INFO:tasks.rados.rados.0.smithi079.stdout:17: left oid 17 (ObjNum 16 snap 0 seq_num 16) 2024-04-24T01:55:32.774 INFO:tasks.rados.rados.0.smithi079.stdout:17: done (15 left) 2024-04-24T01:55:32.774 INFO:tasks.rados.rados.0.smithi079.stdout:37: write initial oid 37 2024-04-24T01:55:32.774 INFO:tasks.rados.rados.0.smithi079.stdout:37: seq_num 36 ranges {605052=774376,1923587=633336} 2024-04-24T01:55:32.780 INFO:tasks.rados.rados.0.smithi079.stdout:37: writing smithi07929762-37 from 605052 to 1379428 tid 1 2024-04-24T01:55:32.789 INFO:tasks.rados.rados.0.smithi079.stdout:37: writing smithi07929762-37 from 1923587 to 2556923 tid 2 2024-04-24T01:55:32.789 INFO:tasks.rados.rados.0.smithi079.stdout: waiting on 16 2024-04-24T01:55:32.789 INFO:tasks.rados.rados.0.smithi079.stdout:29: finishing write tid 1 to smithi07929762-29 2024-04-24T01:55:32.789 INFO:tasks.rados.rados.0.smithi079.stdout:25: finishing write tid 3 to smithi07929762-25 2024-04-24T01:55:32.790 INFO:tasks.rados.rados.0.smithi079.stdout:22: finishing write tid 3 to smithi07929762-22 2024-04-24T01:55:32.792 INFO:tasks.rados.rados.0.smithi079.stdout:25: finishing write tid 4 to smithi07929762-25 2024-04-24T01:55:32.792 INFO:tasks.rados.rados.0.smithi079.stdout:25: oid 25 updating version 0 to 6 2024-04-24T01:55:32.792 INFO:tasks.rados.rados.0.smithi079.stdout:25: oid 25 version 6 is already newer than 4 2024-04-24T01:55:32.792 INFO:tasks.rados.rados.0.smithi079.stdout:25: oid 25 version 6 is already newer than 6 2024-04-24T01:55:32.792 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 25 v 6 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-04-24T01:55:32.792 INFO:tasks.rados.rados.0.smithi079.stdout:25: left oid 25 (ObjNum 24 snap 0 seq_num 24) 2024-04-24T01:55:32.792 INFO:tasks.rados.rados.0.smithi079.stdout:25: done (15 left) 2024-04-24T01:55:32.793 INFO:tasks.rados.rados.0.smithi079.stdout:38: write initial oid 38 2024-04-24T01:55:32.793 INFO:tasks.rados.rados.0.smithi079.stdout:38: seq_num 37 ranges {621859=449625,1865716=756143} 2024-04-24T01:55:32.798 INFO:tasks.rados.rados.0.smithi079.stdout:38: writing smithi07929762-38 from 621859 to 1071484 tid 1 2024-04-24T01:55:32.811 INFO:tasks.rados.rados.0.smithi079.stdout:38: writing smithi07929762-38 from 1865716 to 2621859 tid 2 2024-04-24T01:55:32.811 INFO:tasks.rados.rados.0.smithi079.stdout: waiting on 16 2024-04-24T01:55:32.811 INFO:tasks.rados.rados.0.smithi079.stdout:34: finishing write tid 1 to smithi07929762-34 2024-04-24T01:55:32.811 INFO:tasks.rados.rados.0.smithi079.stdout:22: finishing write tid 4 to smithi07929762-22 2024-04-24T01:55:32.811 INFO:tasks.rados.rados.0.smithi079.stdout:22: oid 22 updating version 0 to 1 2024-04-24T01:55:32.811 INFO:tasks.rados.rados.0.smithi079.stdout:22: oid 22 updating version 1 to 3 2024-04-24T01:55:32.811 INFO:tasks.rados.rados.0.smithi079.stdout:22: oid 22 version 3 is already newer than 3 2024-04-24T01:55:32.811 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 22 v 3 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-04-24T01:55:32.811 INFO:tasks.rados.rados.0.smithi079.stdout:22: left oid 22 (ObjNum 21 snap 0 seq_num 21) 2024-04-24T01:55:32.811 INFO:tasks.rados.rados.0.smithi079.stdout:19: finishing write tid 2 to smithi07929762-19 2024-04-24T01:55:32.812 INFO:tasks.rados.rados.0.smithi079.stdout:19: finishing write tid 3 to smithi07929762-19 2024-04-24T01:55:32.812 INFO:tasks.rados.rados.0.smithi079.stdout:33: finishing write tid 1 to smithi07929762-33 2024-04-24T01:55:32.812 INFO:tasks.rados.rados.0.smithi079.stdout:22: done (15 left) 2024-04-24T01:55:32.812 INFO:tasks.rados.rados.0.smithi079.stdout:39: write initial oid 39 2024-04-24T01:55:32.812 INFO:tasks.rados.rados.0.smithi079.stdout:39: seq_num 38 ranges {638666=524874,1924198=637005} 2024-04-24T01:55:32.819 INFO:tasks.rados.rados.0.smithi079.stdout:39: writing smithi07929762-39 from 638666 to 1163540 tid 1 2024-04-24T01:55:32.830 INFO:tasks.rados.rados.0.smithi079.stdout:39: writing smithi07929762-39 from 1924198 to 2561203 tid 2 2024-04-24T01:55:32.830 INFO:tasks.rados.rados.0.smithi079.stdout: waiting on 16 2024-04-24T01:55:32.830 INFO:tasks.rados.rados.0.smithi079.stdout:19: finishing write tid 4 to smithi07929762-19 2024-04-24T01:55:32.830 INFO:tasks.rados.rados.0.smithi079.stdout:19: oid 19 updating version 0 to 1 2024-04-24T01:55:32.830 INFO:tasks.rados.rados.0.smithi079.stdout:19: oid 19 updating version 1 to 3 2024-04-24T01:55:32.830 INFO:tasks.rados.rados.0.smithi079.stdout:19: oid 19 version 3 is already newer than 3 2024-04-24T01:55:32.830 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 19 v 3 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-04-24T01:55:32.830 INFO:tasks.rados.rados.0.smithi079.stdout:19: left oid 19 (ObjNum 18 snap 0 seq_num 18) 2024-04-24T01:55:32.831 INFO:tasks.rados.rados.0.smithi079.stdout:19: done (15 left) 2024-04-24T01:55:32.831 INFO:tasks.rados.rados.0.smithi079.stdout:40: write initial oid 40 2024-04-24T01:55:32.831 INFO:tasks.rados.rados.0.smithi079.stdout:40: seq_num 39 ranges {655473=716476,2099033=556440} 2024-04-24T01:55:32.841 INFO:tasks.rados.rados.0.smithi079.stdout:40: writing smithi07929762-40 from 655473 to 1371949 tid 1 2024-04-24T01:55:32.851 INFO:tasks.rados.rados.0.smithi079.stdout:40: writing smithi07929762-40 from 2099033 to 2655473 tid 2 2024-04-24T01:55:32.851 INFO:tasks.rados.rados.0.smithi079.stdout: waiting on 16 2024-04-24T01:55:32.851 INFO:tasks.rados.rados.0.smithi079.stdout:30: finishing write tid 1 to smithi07929762-30 2024-04-24T01:55:32.851 INFO:tasks.rados.rados.0.smithi079.stdout:32: finishing write tid 1 to smithi07929762-32 2024-04-24T01:55:32.852 INFO:tasks.rados.rados.0.smithi079.stdout:18: finishing write tid 2 to smithi07929762-18 2024-04-24T01:55:32.852 INFO:tasks.rados.rados.0.smithi079.stdout:28: finishing write tid 2 to smithi07929762-28 2024-04-24T01:55:32.852 INFO:tasks.rados.rados.0.smithi079.stdout:34: finishing write tid 2 to smithi07929762-34 2024-04-24T01:55:32.852 INFO:tasks.rados.rados.0.smithi079.stdout:28: finishing write tid 3 to smithi07929762-28 2024-04-24T01:55:32.853 INFO:tasks.rados.rados.0.smithi079.stdout:28: finishing write tid 4 to smithi07929762-28 2024-04-24T01:55:32.854 INFO:tasks.rados.rados.0.smithi079.stdout:28: oid 28 updating version 0 to 9 2024-04-24T01:55:32.854 INFO:tasks.rados.rados.0.smithi079.stdout:28: oid 28 version 9 is already newer than 7 2024-04-24T01:55:32.854 INFO:tasks.rados.rados.0.smithi079.stdout:28: oid 28 version 9 is already newer than 9 2024-04-24T01:55:32.854 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 28 v 9 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-04-24T01:55:32.854 INFO:tasks.rados.rados.0.smithi079.stdout:28: left oid 28 (ObjNum 27 snap 0 seq_num 27) 2024-04-24T01:55:32.854 INFO:tasks.rados.rados.0.smithi079.stdout:28: done (15 left) 2024-04-24T01:55:32.854 INFO:tasks.rados.rados.0.smithi079.stdout:41: write initial oid 41 2024-04-24T01:55:32.854 INFO:tasks.rados.rados.0.smithi079.stdout:41: seq_num 40 ranges {672280=791725,2157515=514765} 2024-04-24T01:55:32.864 INFO:tasks.rados.rados.0.smithi079.stdout:41: writing smithi07929762-41 from 672280 to 1464005 tid 1 2024-04-24T01:55:32.873 INFO:tasks.rados.rados.0.smithi079.stdout:41: writing smithi07929762-41 from 2157515 to 2672280 tid 2 2024-04-24T01:55:32.873 INFO:tasks.rados.rados.0.smithi079.stdout:38: finishing write tid 1 to smithi07929762-38 2024-04-24T01:55:32.873 INFO:tasks.rados.rados.0.smithi079.stdout:18: finishing write tid 3 to smithi07929762-18 2024-04-24T01:55:32.873 INFO:tasks.rados.rados.0.smithi079.stdout:18: finishing write tid 4 to smithi07929762-18 2024-04-24T01:55:32.874 INFO:tasks.rados.rados.0.smithi079.stdout:18: oid 18 updating version 0 to 10 2024-04-24T01:55:32.874 INFO:tasks.rados.rados.0.smithi079.stdout:18: oid 18 updating version 10 to 12 2024-04-24T01:55:32.874 INFO:tasks.rados.rados.0.smithi079.stdout:18: oid 18 version 12 is already newer than 12 2024-04-24T01:55:32.874 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 18 v 12 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-04-24T01:55:32.874 INFO:tasks.rados.rados.0.smithi079.stdout:18: left oid 18 (ObjNum 17 snap 0 seq_num 17) 2024-04-24T01:55:32.874 INFO:tasks.rados.rados.0.smithi079.stdout:34: finishing write tid 3 to smithi07929762-34 2024-04-24T01:55:32.874 INFO:tasks.rados.rados.0.smithi079.stdout:18: done (15 left) 2024-04-24T01:55:32.874 INFO:tasks.rados.rados.0.smithi079.stdout:42: write initial oid 42 2024-04-24T01:55:32.874 INFO:tasks.rados.rados.0.smithi079.stdout:42: seq_num 41 ranges {689087=466974,1699644=784332} 2024-04-24T01:55:32.881 INFO:tasks.rados.rados.0.smithi079.stdout:42: writing smithi07929762-42 from 689087 to 1156061 tid 1 2024-04-24T01:55:32.889 INFO:tasks.rados.rados.0.smithi079.stdout:42: writing smithi07929762-42 from 1699644 to 2483976 tid 2 2024-04-24T01:55:32.890 INFO:tasks.rados.rados.0.smithi079.stdout: waiting on 16 2024-04-24T01:55:32.890 INFO:tasks.rados.rados.0.smithi079.stdout:34: finishing write tid 4 to smithi07929762-34 2024-04-24T01:55:32.890 INFO:tasks.rados.rados.0.smithi079.stdout:34: oid 34 updating version 0 to 6 2024-04-24T01:55:32.890 INFO:tasks.rados.rados.0.smithi079.stdout:34: oid 34 version 6 is already newer than 4 2024-04-24T01:55:32.890 INFO:tasks.rados.rados.0.smithi079.stdout:34: oid 34 version 6 is already newer than 6 2024-04-24T01:55:32.890 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 34 v 6 (ObjNum 33 snap 0 seq_num 33) dirty exists 2024-04-24T01:55:32.890 INFO:tasks.rados.rados.0.smithi079.stdout:34: left oid 34 (ObjNum 33 snap 0 seq_num 33) 2024-04-24T01:55:32.890 INFO:tasks.rados.rados.0.smithi079.stdout:34: done (15 left) 2024-04-24T01:55:32.890 INFO:tasks.rados.rados.0.smithi079.stdout:43: write initial oid 43 2024-04-24T01:55:32.890 INFO:tasks.rados.rados.0.smithi079.stdout:43: seq_num 42 ranges {705894=542223,1758126=644343} 2024-04-24T01:55:32.897 INFO:tasks.rados.rados.0.smithi079.stdout:43: writing smithi07929762-43 from 705894 to 1248117 tid 1 2024-04-24T01:55:32.904 INFO:tasks.rados.rados.0.smithi079.stdout:43: writing smithi07929762-43 from 1758126 to 2402469 tid 2 2024-04-24T01:55:32.904 INFO:tasks.rados.rados.0.smithi079.stdout: waiting on 16 2024-04-24T01:55:32.904 INFO:tasks.rados.rados.0.smithi079.stdout:39: finishing write tid 1 to smithi07929762-39 2024-04-24T01:55:32.905 INFO:tasks.rados.rados.0.smithi079.stdout:32: finishing write tid 2 to smithi07929762-32 2024-04-24T01:55:32.905 INFO:tasks.rados.rados.0.smithi079.stdout:36: finishing write tid 1 to smithi07929762-36 2024-04-24T01:55:32.913 INFO:tasks.rados.rados.0.smithi079.stdout:29: finishing write tid 2 to smithi07929762-29 2024-04-24T01:55:32.914 INFO:tasks.rados.rados.0.smithi079.stdout:31: finishing write tid 1 to smithi07929762-31 2024-04-24T01:55:32.915 INFO:tasks.rados.rados.0.smithi079.stdout:29: finishing write tid 3 to smithi07929762-29 2024-04-24T01:55:32.917 INFO:tasks.rados.rados.0.smithi079.stdout:26: finishing write tid 1 to smithi07929762-26 2024-04-24T01:55:32.925 INFO:tasks.rados.rados.0.smithi079.stdout:29: finishing write tid 4 to smithi07929762-29 2024-04-24T01:55:32.925 INFO:tasks.rados.rados.0.smithi079.stdout:29: oid 29 updating version 0 to 9 2024-04-24T01:55:32.925 INFO:tasks.rados.rados.0.smithi079.stdout:29: oid 29 version 9 is already newer than 9 2024-04-24T01:55:32.925 INFO:tasks.rados.rados.0.smithi079.stdout:29: oid 29 version 9 is already newer than 7 2024-04-24T01:55:32.925 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 29 v 9 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-04-24T01:55:32.925 INFO:tasks.rados.rados.0.smithi079.stdout:29: left oid 29 (ObjNum 28 snap 0 seq_num 28) 2024-04-24T01:55:32.925 INFO:tasks.rados.rados.0.smithi079.stdout:29: done (15 left) 2024-04-24T01:55:32.925 INFO:tasks.rados.rados.0.smithi079.stdout:44: write initial oid 44 2024-04-24T01:55:32.925 INFO:tasks.rados.rados.0.smithi079.stdout:44: seq_num 43 ranges {722701=617472,1816608=788001} 2024-04-24T01:55:32.935 INFO:tasks.rados.rados.0.smithi079.stdout:44: writing smithi07929762-44 from 722701 to 1340173 tid 1 2024-04-24T01:55:32.947 INFO:tasks.rados.rados.0.smithi079.stdout:44: writing smithi07929762-44 from 1816608 to 2604609 tid 2 2024-04-24T01:55:32.947 INFO:tasks.rados.rados.0.smithi079.stdout: waiting on 16 2024-04-24T01:55:32.947 INFO:tasks.rados.rados.0.smithi079.stdout:32: finishing write tid 3 to smithi07929762-32 2024-04-24T01:55:32.947 INFO:tasks.rados.rados.0.smithi079.stdout:32: finishing write tid 4 to smithi07929762-32 2024-04-24T01:55:32.947 INFO:tasks.rados.rados.0.smithi079.stdout:32: oid 32 updating version 0 to 1 2024-04-24T01:55:32.948 INFO:tasks.rados.rados.0.smithi079.stdout:32: oid 32 updating version 1 to 3 2024-04-24T01:55:32.948 INFO:tasks.rados.rados.0.smithi079.stdout:32: oid 32 version 3 is already newer than 3 2024-04-24T01:55:32.948 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 32 v 3 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-04-24T01:55:32.948 INFO:tasks.rados.rados.0.smithi079.stdout:32: left oid 32 (ObjNum 31 snap 0 seq_num 31) 2024-04-24T01:55:32.948 INFO:tasks.rados.rados.0.smithi079.stdout:38: finishing write tid 2 to smithi07929762-38 2024-04-24T01:55:32.948 INFO:tasks.rados.rados.0.smithi079.stdout:24: finishing write tid 1 to smithi07929762-24 2024-04-24T01:55:32.949 INFO:tasks.rados.rados.0.smithi079.stdout:32: done (15 left) 2024-04-24T01:55:32.949 INFO:tasks.rados.rados.0.smithi079.stdout:45: write initial oid 45 2024-04-24T01:55:32.949 INFO:tasks.rados.rados.0.smithi079.stdout:45: seq_num 44 ranges {739508=692721,1875090=648012} 2024-04-24T01:55:32.955 INFO:tasks.rados.rados.0.smithi079.stdout:45: writing smithi07929762-45 from 739508 to 1432229 tid 1 2024-04-24T01:55:32.962 INFO:tasks.rados.rados.0.smithi079.stdout:45: writing smithi07929762-45 from 1875090 to 2523102 tid 2 2024-04-24T01:55:32.962 INFO:tasks.rados.rados.0.smithi079.stdout: waiting on 16 2024-04-24T01:55:32.962 INFO:tasks.rados.rados.0.smithi079.stdout:38: finishing write tid 3 to smithi07929762-38 2024-04-24T01:55:32.962 INFO:tasks.rados.rados.0.smithi079.stdout:38: finishing write tid 4 to smithi07929762-38 2024-04-24T01:55:32.962 INFO:tasks.rados.rados.0.smithi079.stdout:38: oid 38 updating version 0 to 13 2024-04-24T01:55:32.962 INFO:tasks.rados.rados.0.smithi079.stdout:38: oid 38 updating version 13 to 15 2024-04-24T01:55:32.963 INFO:tasks.rados.rados.0.smithi079.stdout:38: oid 38 version 15 is already newer than 15 2024-04-24T01:55:32.963 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 38 v 15 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-04-24T01:55:32.963 INFO:tasks.rados.rados.0.smithi079.stdout:38: left oid 38 (ObjNum 37 snap 0 seq_num 37) 2024-04-24T01:55:32.963 INFO:tasks.rados.rados.0.smithi079.stdout:33: finishing write tid 2 to smithi07929762-33 2024-04-24T01:55:32.963 INFO:tasks.rados.rados.0.smithi079.stdout:38: done (15 left) 2024-04-24T01:55:32.963 INFO:tasks.rados.rados.0.smithi079.stdout:46: write initial oid 46 2024-04-24T01:55:32.963 INFO:tasks.rados.rados.0.smithi079.stdout:46: seq_num 45 ranges {756315=767970,1933572=791670} 2024-04-24T01:55:32.972 INFO:tasks.rados.rados.0.smithi079.stdout:46: writing smithi07929762-46 from 756315 to 1524285 tid 1 2024-04-24T01:55:32.981 INFO:tasks.rados.rados.0.smithi079.stdout:46: writing smithi07929762-46 from 1933572 to 2725242 tid 2 2024-04-24T01:55:32.981 INFO:tasks.rados.rados.0.smithi079.stdout: waiting on 16 2024-04-24T01:55:32.981 INFO:tasks.rados.rados.0.smithi079.stdout:33: finishing write tid 3 to smithi07929762-33 2024-04-24T01:55:32.981 INFO:tasks.rados.rados.0.smithi079.stdout:31: finishing write tid 2 to smithi07929762-31 2024-04-24T01:55:32.981 INFO:tasks.rados.rados.0.smithi079.stdout:30: finishing write tid 2 to smithi07929762-30 2024-04-24T01:55:32.982 INFO:tasks.rados.rados.0.smithi079.stdout:39: finishing write tid 2 to smithi07929762-39 2024-04-24T01:55:32.982 INFO:tasks.rados.rados.0.smithi079.stdout:33: finishing write tid 4 to smithi07929762-33 2024-04-24T01:55:32.982 INFO:tasks.rados.rados.0.smithi079.stdout:33: oid 33 updating version 0 to 9 2024-04-24T01:55:32.982 INFO:tasks.rados.rados.0.smithi079.stdout:33: oid 33 version 9 is already newer than 7 2024-04-24T01:55:32.982 INFO:tasks.rados.rados.0.smithi079.stdout:33: oid 33 version 9 is already newer than 9 2024-04-24T01:55:32.982 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 33 v 9 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-04-24T01:55:32.982 INFO:tasks.rados.rados.0.smithi079.stdout:33: left oid 33 (ObjNum 32 snap 0 seq_num 32) 2024-04-24T01:55:32.982 INFO:tasks.rados.rados.0.smithi079.stdout:31: finishing write tid 3 to smithi07929762-31 2024-04-24T01:55:32.982 INFO:tasks.rados.rados.0.smithi079.stdout:30: finishing write tid 3 to smithi07929762-30 2024-04-24T01:55:32.982 INFO:tasks.rados.rados.0.smithi079.stdout:31: finishing write tid 4 to smithi07929762-31 2024-04-24T01:55:32.982 INFO:tasks.rados.rados.0.smithi079.stdout:31: oid 31 updating version 0 to 15 2024-04-24T01:55:32.982 INFO:tasks.rados.rados.0.smithi079.stdout:31: oid 31 version 15 is already newer than 15 2024-04-24T01:55:32.982 INFO:tasks.rados.rados.0.smithi079.stdout:31: oid 31 version 15 is already newer than 13 2024-04-24T01:55:32.983 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 31 v 15 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-04-24T01:55:32.983 INFO:tasks.rados.rados.0.smithi079.stdout:31: left oid 31 (ObjNum 30 snap 0 seq_num 30) 2024-04-24T01:55:32.983 INFO:tasks.rados.rados.0.smithi079.stdout:39: finishing write tid 3 to smithi07929762-39 2024-04-24T01:55:32.983 INFO:tasks.rados.rados.0.smithi079.stdout:31: done (15 left) 2024-04-24T01:55:32.983 INFO:tasks.rados.rados.0.smithi079.stdout:33: done (14 left) 2024-04-24T01:55:32.983 INFO:tasks.rados.rados.0.smithi079.stdout:47: write initial oid 47 2024-04-24T01:55:32.983 INFO:tasks.rados.rados.0.smithi079.stdout:47: seq_num 46 ranges {773122=559572,1992054=651681} 2024-04-24T01:55:32.989 INFO:tasks.rados.rados.0.smithi079.stdout:47: writing smithi07929762-47 from 773122 to 1332694 tid 1 2024-04-24T01:55:33.000 INFO:tasks.rados.rados.0.smithi079.stdout:47: writing smithi07929762-47 from 1992054 to 2643735 tid 2 2024-04-24T01:55:33.000 INFO:tasks.rados.rados.0.smithi079.stdout:48: write initial oid 48 2024-04-24T01:55:33.001 INFO:tasks.rados.rados.0.smithi079.stdout:48: seq_num 47 ranges {789929=634821,2050536=739393} 2024-04-24T01:55:33.008 INFO:tasks.rados.rados.0.smithi079.stdout:48: writing smithi07929762-48 from 789929 to 1424750 tid 1 2024-04-24T01:55:33.018 INFO:tasks.rados.rados.0.smithi079.stdout:48: writing smithi07929762-48 from 2050536 to 2789929 tid 2 2024-04-24T01:55:33.018 INFO:tasks.rados.rados.0.smithi079.stdout: waiting on 16 2024-04-24T01:55:33.018 INFO:tasks.rados.rados.0.smithi079.stdout:39: finishing write tid 4 to smithi07929762-39 2024-04-24T01:55:33.018 INFO:tasks.rados.rados.0.smithi079.stdout:39: oid 39 updating version 0 to 6 2024-04-24T01:55:33.018 INFO:tasks.rados.rados.0.smithi079.stdout:39: oid 39 version 6 is already newer than 4 2024-04-24T01:55:33.018 INFO:tasks.rados.rados.0.smithi079.stdout:39: oid 39 version 6 is already newer than 6 2024-04-24T01:55:33.018 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 39 v 6 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-04-24T01:55:33.018 INFO:tasks.rados.rados.0.smithi079.stdout:39: left oid 39 (ObjNum 38 snap 0 seq_num 38) 2024-04-24T01:55:33.018 INFO:tasks.rados.rados.0.smithi079.stdout:39: done (15 left) 2024-04-24T01:55:33.018 INFO:tasks.rados.rados.0.smithi079.stdout:49: write initial oid 49 2024-04-24T01:55:33.018 INFO:tasks.rados.rados.0.smithi079.stdout:49: seq_num 48 ranges {406736=710070,1709018=655350} 2024-04-24T01:55:33.026 INFO:tasks.rados.rados.0.smithi079.stdout:49: writing smithi07929762-49 from 406736 to 1116806 tid 1 2024-04-24T01:55:33.037 INFO:tasks.rados.rados.0.smithi079.stdout:49: writing smithi07929762-49 from 1709018 to 2364368 tid 2 2024-04-24T01:55:33.037 INFO:tasks.rados.rados.0.smithi079.stdout: waiting on 16 2024-04-24T01:55:33.037 INFO:tasks.rados.rados.0.smithi079.stdout:30: finishing write tid 4 to smithi07929762-30 2024-04-24T01:55:33.037 INFO:tasks.rados.rados.0.smithi079.stdout:30: oid 30 updating version 0 to 3 2024-04-24T01:55:33.037 INFO:tasks.rados.rados.0.smithi079.stdout:30: oid 30 version 3 is already newer than 1 2024-04-24T01:55:33.037 INFO:tasks.rados.rados.0.smithi079.stdout:30: oid 30 version 3 is already newer than 3 2024-04-24T01:55:33.037 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 30 v 3 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-04-24T01:55:33.037 INFO:tasks.rados.rados.0.smithi079.stdout:30: left oid 30 (ObjNum 29 snap 0 seq_num 29) 2024-04-24T01:55:33.037 INFO:tasks.rados.rados.0.smithi079.stdout:30: done (15 left) 2024-04-24T01:55:33.038 INFO:tasks.rados.rados.0.smithi079.stdout:50: write initial oid 50 2024-04-24T01:55:33.038 INFO:tasks.rados.rados.0.smithi079.stdout:50: seq_num 49 ranges {423543=785319,1767500=799008} 2024-04-24T01:55:33.043 INFO:tasks.rados.rados.0.smithi079.stdout:50: writing smithi07929762-50 from 423543 to 1208862 tid 1 2024-04-24T01:55:33.052 INFO:tasks.rados.rados.0.smithi079.stdout:50: writing smithi07929762-50 from 1767500 to 2566508 tid 2 2024-04-24T01:55:33.052 INFO:tasks.rados.rados.0.smithi079.stdout: waiting on 16 2024-04-24T01:55:33.052 INFO:tasks.rados.rados.0.smithi079.stdout:36: finishing write tid 2 to smithi07929762-36 2024-04-24T01:55:33.053 INFO:tasks.rados.rados.0.smithi079.stdout:36: finishing write tid 3 to smithi07929762-36 2024-04-24T01:55:33.053 INFO:tasks.rados.rados.0.smithi079.stdout:36: finishing write tid 4 to smithi07929762-36 2024-04-24T01:55:33.053 INFO:tasks.rados.rados.0.smithi079.stdout:36: oid 36 updating version 0 to 4 2024-04-24T01:55:33.053 INFO:tasks.rados.rados.0.smithi079.stdout:36: oid 36 updating version 4 to 6 2024-04-24T01:55:33.053 INFO:tasks.rados.rados.0.smithi079.stdout:36: oid 36 version 6 is already newer than 6 2024-04-24T01:55:33.053 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 36 v 6 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-04-24T01:55:33.053 INFO:tasks.rados.rados.0.smithi079.stdout:36: left oid 36 (ObjNum 35 snap 0 seq_num 35) 2024-04-24T01:55:33.053 INFO:tasks.rados.rados.0.smithi079.stdout:41: finishing write tid 1 to smithi07929762-41 2024-04-24T01:55:33.053 INFO:tasks.rados.rados.0.smithi079.stdout:37: finishing write tid 1 to smithi07929762-37 2024-04-24T01:55:33.053 INFO:tasks.rados.rados.0.smithi079.stdout:26: finishing write tid 2 to smithi07929762-26 2024-04-24T01:55:33.053 INFO:tasks.rados.rados.0.smithi079.stdout:42: finishing write tid 1 to smithi07929762-42 2024-04-24T01:55:33.053 INFO:tasks.rados.rados.0.smithi079.stdout:45: finishing write tid 1 to smithi07929762-45 2024-04-24T01:55:33.053 INFO:tasks.rados.rados.0.smithi079.stdout:24: finishing write tid 2 to smithi07929762-24 2024-04-24T01:55:33.053 INFO:tasks.rados.rados.0.smithi079.stdout:36: done (15 left) 2024-04-24T01:55:33.053 INFO:tasks.rados.rados.0.smithi079.stdout:51: write (excl) oid 2 current snap is 0 2024-04-24T01:55:33.053 INFO:tasks.rados.rados.0.smithi079.stdout:51: seq_num 50 ranges {440350=460568,1309629=542666,2323973=516377} 2024-04-24T01:55:33.058 INFO:tasks.rados.rados.0.smithi079.stdout:51: writing smithi07929762-2 from 440350 to 900918 tid 1 2024-04-24T01:55:33.065 INFO:tasks.rados.rados.0.smithi079.stdout:51: writing smithi07929762-2 from 1309629 to 1852295 tid 2 2024-04-24T01:55:33.072 INFO:tasks.rados.rados.0.smithi079.stdout:51: writing smithi07929762-2 from 2323973 to 2840350 tid 3 2024-04-24T01:55:33.073 INFO:tasks.rados.rados.0.smithi079.stdout: waiting on 16 2024-04-24T01:55:33.073 INFO:tasks.rados.rados.0.smithi079.stdout:26: finishing write tid 3 to smithi07929762-26 2024-04-24T01:55:33.074 INFO:tasks.rados.rados.0.smithi079.stdout:26: finishing write tid 4 to smithi07929762-26 2024-04-24T01:55:33.074 INFO:tasks.rados.rados.0.smithi079.stdout:26: oid 26 updating version 0 to 9 2024-04-24T01:55:33.074 INFO:tasks.rados.rados.0.smithi079.stdout:26: oid 26 version 9 is already newer than 9 2024-04-24T01:55:33.074 INFO:tasks.rados.rados.0.smithi079.stdout:26: oid 26 version 9 is already newer than 7 2024-04-24T01:55:33.074 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 26 v 9 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-04-24T01:55:33.074 INFO:tasks.rados.rados.0.smithi079.stdout:26: left oid 26 (ObjNum 25 snap 0 seq_num 25) 2024-04-24T01:55:33.074 INFO:tasks.rados.rados.0.smithi079.stdout:24: finishing write tid 3 to smithi07929762-24 2024-04-24T01:55:33.074 INFO:tasks.rados.rados.0.smithi079.stdout:44: finishing write tid 1 to smithi07929762-44 2024-04-24T01:55:33.074 INFO:tasks.rados.rados.0.smithi079.stdout:26: done (15 left) 2024-04-24T01:55:33.074 INFO:tasks.rados.rados.0.smithi079.stdout:52: write (excl) oid 14 current snap is 0 2024-04-24T01:55:33.074 INFO:tasks.rados.rados.0.smithi079.stdout:52: seq_num 51 ranges {457157=535817,1768111=402677} 2024-04-24T01:55:33.080 INFO:tasks.rados.rados.0.smithi079.stdout:52: writing smithi07929762-14 from 457157 to 992974 tid 1 2024-04-24T01:55:33.088 INFO:tasks.rados.rados.0.smithi079.stdout:52: writing smithi07929762-14 from 1768111 to 2170788 tid 2 2024-04-24T01:55:33.088 INFO:tasks.rados.rados.0.smithi079.stdout: waiting on 16 2024-04-24T01:55:33.088 INFO:tasks.rados.rados.0.smithi079.stdout:24: finishing write tid 4 to smithi07929762-24 2024-04-24T01:55:33.088 INFO:tasks.rados.rados.0.smithi079.stdout:24: oid 24 updating version 0 to 6 2024-04-24T01:55:33.088 INFO:tasks.rados.rados.0.smithi079.stdout:24: oid 24 version 6 is already newer than 6 2024-04-24T01:55:33.088 INFO:tasks.rados.rados.0.smithi079.stdout:24: oid 24 version 6 is already newer than 4 2024-04-24T01:55:33.088 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 24 v 6 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-04-24T01:55:33.088 INFO:tasks.rados.rados.0.smithi079.stdout:24: left oid 24 (ObjNum 23 snap 0 seq_num 23) 2024-04-24T01:55:33.088 INFO:tasks.rados.rados.0.smithi079.stdout:46: finishing write tid 1 to smithi07929762-46 2024-04-24T01:55:33.088 INFO:tasks.rados.rados.0.smithi079.stdout:24: done (15 left) 2024-04-24T01:55:33.088 INFO:tasks.rados.rados.0.smithi079.stdout:53: read oid 20 snap -1 2024-04-24T01:55:33.088 INFO:tasks.rados.rados.0.smithi079.stdout:53: expect (ObjNum 19 snap 0 seq_num 19) 2024-04-24T01:55:33.088 INFO:tasks.rados.rados.0.smithi079.stdout: waiting on 16 2024-04-24T01:55:33.089 INFO:tasks.rados.rados.0.smithi079.stdout:40: finishing write tid 1 to smithi07929762-40 2024-04-24T01:55:33.108 INFO:tasks.rados.rados.0.smithi079.stdout:43: finishing write tid 1 to smithi07929762-43 2024-04-24T01:55:33.110 INFO:tasks.rados.rados.0.smithi079.stdout:48: finishing write tid 1 to smithi07929762-48 2024-04-24T01:55:33.118 INFO:tasks.rados.rados.0.smithi079.stdout:45: finishing write tid 2 to smithi07929762-45 2024-04-24T01:55:33.118 INFO:tasks.rados.rados.0.smithi079.stdout:37: finishing write tid 2 to smithi07929762-37 2024-04-24T01:55:33.125 INFO:tasks.rados.rados.0.smithi079.stdout:37: finishing write tid 3 to smithi07929762-37 2024-04-24T01:55:33.129 INFO:tasks.rados.rados.0.smithi079.stdout:42: finishing write tid 2 to smithi07929762-42 2024-04-24T01:55:33.134 INFO:tasks.rados.rados.0.smithi079.stdout:45: finishing write tid 3 to smithi07929762-45 2024-04-24T01:55:33.136 INFO:tasks.rados.rados.0.smithi079.stdout:45: finishing write tid 4 to smithi07929762-45 2024-04-24T01:55:33.136 INFO:tasks.rados.rados.0.smithi079.stdout:45: oid 45 updating version 0 to 10 2024-04-24T01:55:33.136 INFO:tasks.rados.rados.0.smithi079.stdout:45: oid 45 updating version 10 to 12 2024-04-24T01:55:33.136 INFO:tasks.rados.rados.0.smithi079.stdout:45: oid 45 version 12 is already newer than 12 2024-04-24T01:55:33.136 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 45 v 12 (ObjNum 44 snap 0 seq_num 44) dirty exists 2024-04-24T01:55:33.136 INFO:tasks.rados.rados.0.smithi079.stdout:45: left oid 45 (ObjNum 44 snap 0 seq_num 44) 2024-04-24T01:55:33.136 INFO:tasks.rados.rados.0.smithi079.stdout:45: done (15 left) 2024-04-24T01:55:33.136 INFO:tasks.rados.rados.0.smithi079.stdout:54: delete oid 23 current snap is 0 2024-04-24T01:55:33.143 INFO:tasks.rados.rados.0.smithi079.stdout:42: finishing write tid 3 to smithi07929762-42 2024-04-24T01:55:33.149 INFO:tasks.rados.rados.0.smithi079.stdout:37: finishing write tid 4 to smithi07929762-37 2024-04-24T01:55:33.149 INFO:tasks.rados.rados.0.smithi079.stdout:37: oid 37 updating version 0 to 16 2024-04-24T01:55:33.149 INFO:tasks.rados.rados.0.smithi079.stdout:37: oid 37 updating version 16 to 18 2024-04-24T01:55:33.149 INFO:tasks.rados.rados.0.smithi079.stdout:37: oid 37 version 18 is already newer than 18 2024-04-24T01:55:33.149 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 37 v 18 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-04-24T01:55:33.149 INFO:tasks.rados.rados.0.smithi079.stdout:37: left oid 37 (ObjNum 36 snap 0 seq_num 36) 2024-04-24T01:55:33.151 INFO:tasks.rados.rados.0.smithi079.stdout:42: finishing write tid 4 to smithi07929762-42 2024-04-24T01:55:33.151 INFO:tasks.rados.rados.0.smithi079.stdout:42: oid 42 updating version 0 to 12 2024-04-24T01:55:33.151 INFO:tasks.rados.rados.0.smithi079.stdout:42: oid 42 version 12 is already newer than 12 2024-04-24T01:55:33.151 INFO:tasks.rados.rados.0.smithi079.stdout:42: oid 42 version 12 is already newer than 10 2024-04-24T01:55:33.151 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 42 v 12 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-04-24T01:55:33.151 INFO:tasks.rados.rados.0.smithi079.stdout:42: left oid 42 (ObjNum 41 snap 0 seq_num 41) 2024-04-24T01:55:33.156 INFO:tasks.rados.rados.0.smithi079.stdout:46: finishing write tid 2 to smithi07929762-46 2024-04-24T01:55:33.166 INFO:tasks.rados.rados.0.smithi079.stdout:44: finishing write tid 2 to smithi07929762-44 2024-04-24T01:55:33.166 INFO:tasks.rados.rados.0.smithi079.stdout:46: finishing write tid 3 to smithi07929762-46 2024-04-24T01:55:33.171 INFO:tasks.rados.rados.0.smithi079.stdout:46: finishing write tid 4 to smithi07929762-46 2024-04-24T01:55:33.171 INFO:tasks.rados.rados.0.smithi079.stdout:46: oid 46 updating version 0 to 9 2024-04-24T01:55:33.171 INFO:tasks.rados.rados.0.smithi079.stdout:46: oid 46 version 9 is already newer than 9 2024-04-24T01:55:33.171 INFO:tasks.rados.rados.0.smithi079.stdout:46: oid 46 version 9 is already newer than 7 2024-04-24T01:55:33.171 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 46 v 9 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-04-24T01:55:33.171 INFO:tasks.rados.rados.0.smithi079.stdout:46: left oid 46 (ObjNum 45 snap 0 seq_num 45) 2024-04-24T01:55:33.172 INFO:tasks.rados.rados.0.smithi079.stdout:51: finishing write tid 1 to smithi07929762-2 2024-04-24T01:55:33.173 INFO:tasks.rados.rados.0.smithi079.stdout:41: finishing write tid 2 to smithi07929762-41 2024-04-24T01:55:33.181 INFO:tasks.rados.rados.0.smithi079.stdout:44: finishing write tid 3 to smithi07929762-44 2024-04-24T01:55:33.182 INFO:tasks.rados.rados.0.smithi079.stdout:49: finishing write tid 1 to smithi07929762-49 2024-04-24T01:55:33.182 INFO:tasks.rados.rados.0.smithi079.stdout:37: done (15 left) 2024-04-24T01:55:33.182 INFO:tasks.rados.rados.0.smithi079.stdout:42: done (14 left) 2024-04-24T01:55:33.182 INFO:tasks.rados.rados.0.smithi079.stdout:46: done (13 left) 2024-04-24T01:55:33.182 INFO:tasks.rados.rados.0.smithi079.stdout:54: done (12 left) 2024-04-24T01:55:33.182 INFO:tasks.rados.rados.0.smithi079.stdout:55: write oid 10 current snap is 0 2024-04-24T01:55:33.182 INFO:tasks.rados.rados.0.smithi079.stdout:55: seq_num 53 ranges {490771=686315,1885075=406346} 2024-04-24T01:55:33.192 INFO:tasks.rados.rados.0.smithi079.stdout:55: writing smithi07929762-10 from 490771 to 1177086 tid 1 2024-04-24T01:55:33.200 INFO:tasks.rados.rados.0.smithi079.stdout:55: writing smithi07929762-10 from 1885075 to 2291421 tid 2 2024-04-24T01:55:33.200 INFO:tasks.rados.rados.0.smithi079.stdout:56: write oid 25 current snap is 0 2024-04-24T01:55:33.200 INFO:tasks.rados.rados.0.smithi079.stdout:56: seq_num 54 ranges {507578=477917,1543557=550004,2833665=73913} 2024-04-24T01:55:33.208 INFO:tasks.rados.rados.0.smithi079.stdout:56: writing smithi07929762-25 from 507578 to 985495 tid 1 2024-04-24T01:55:33.216 INFO:tasks.rados.rados.0.smithi079.stdout:56: writing smithi07929762-25 from 1543557 to 2093561 tid 2 2024-04-24T01:55:33.221 INFO:tasks.rados.rados.0.smithi079.stdout:56: writing smithi07929762-25 from 2833665 to 2907578 tid 3 2024-04-24T01:55:33.221 INFO:tasks.rados.rados.0.smithi079.stdout:57: write (excl) oid 29 current snap is 0 2024-04-24T01:55:33.221 INFO:tasks.rados.rados.0.smithi079.stdout:44: finishing write tid 4 to smithi07929762-44 2024-04-24T01:55:33.221 INFO:tasks.rados.rados.0.smithi079.stdout:44: oid 44 updating version 0 to 9 2024-04-24T01:55:33.221 INFO:tasks.rados.rados.0.smithi079.stdout:44: oid 44 version 9 is already newer than 7 2024-04-24T01:55:33.221 INFO:tasks.rados.rados.0.smithi079.stdout:44: oid 44 version 9 is already newer than 9 2024-04-24T01:55:33.221 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 44 v 9 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-04-24T01:55:33.221 INFO:tasks.rados.rados.0.smithi079.stdout:44: left oid 44 (ObjNum 43 snap 0 seq_num 43) 2024-04-24T01:55:33.222 INFO:tasks.rados.rados.0.smithi079.stdout:47: finishing write tid 1 to smithi07929762-47 2024-04-24T01:55:33.222 INFO:tasks.rados.rados.0.smithi079.stdout:48: finishing write tid 2 to smithi07929762-48 2024-04-24T01:55:33.222 INFO:tasks.rados.rados.0.smithi079.stdout:40: finishing write tid 2 to smithi07929762-40 2024-04-24T01:55:33.222 INFO:tasks.rados.rados.0.smithi079.stdout:48: finishing write tid 3 to smithi07929762-48 2024-04-24T01:55:33.222 INFO:tasks.rados.rados.0.smithi079.stdout:57: seq_num 55 ranges {524385=553166,1602039=410015,2577441=346944} 2024-04-24T01:55:33.227 INFO:tasks.rados.rados.0.smithi079.stdout:57: writing smithi07929762-29 from 524385 to 1077551 tid 1 2024-04-24T01:55:33.233 INFO:tasks.rados.rados.0.smithi079.stdout:57: writing smithi07929762-29 from 1602039 to 2012054 tid 2 2024-04-24T01:55:33.239 INFO:tasks.rados.rados.0.smithi079.stdout:57: writing smithi07929762-29 from 2577441 to 2924385 tid 3 2024-04-24T01:55:33.239 INFO:tasks.rados.rados.0.smithi079.stdout:44: done (14 left) 2024-04-24T01:55:33.239 INFO:tasks.rados.rados.0.smithi079.stdout:58: read oid 9 snap -1 2024-04-24T01:55:33.239 INFO:tasks.rados.rados.0.smithi079.stdout:58: expect (ObjNum 8 snap 0 seq_num 8) 2024-04-24T01:55:33.239 INFO:tasks.rados.rados.0.smithi079.stdout:59: write (excl) oid 24 current snap is 0 2024-04-24T01:55:33.239 INFO:tasks.rados.rados.0.smithi079.stdout:59: seq_num 56 ranges {541192=628415,1660521=553673,2888511=52681} 2024-04-24T01:55:33.247 INFO:tasks.rados.rados.0.smithi079.stdout:59: writing smithi07929762-24 from 541192 to 1169607 tid 1 2024-04-24T01:55:33.254 INFO:tasks.rados.rados.0.smithi079.stdout:59: writing smithi07929762-24 from 1660521 to 2214194 tid 2 2024-04-24T01:55:33.261 INFO:tasks.rados.rados.0.smithi079.stdout:59: writing smithi07929762-24 from 2888511 to 2941192 tid 3 2024-04-24T01:55:33.261 INFO:tasks.rados.rados.0.smithi079.stdout: waiting on 16 2024-04-24T01:55:33.261 INFO:tasks.rados.rados.0.smithi079.stdout:49: finishing write tid 2 to smithi07929762-49 2024-04-24T01:55:33.261 INFO:tasks.rados.rados.0.smithi079.stdout:48: finishing write tid 4 to smithi07929762-48 2024-04-24T01:55:33.261 INFO:tasks.rados.rados.0.smithi079.stdout:48: oid 48 updating version 0 to 6 2024-04-24T01:55:33.261 INFO:tasks.rados.rados.0.smithi079.stdout:48: oid 48 version 6 is already newer than 4 2024-04-24T01:55:33.261 INFO:tasks.rados.rados.0.smithi079.stdout:48: oid 48 version 6 is already newer than 6 2024-04-24T01:55:33.261 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 48 v 6 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-04-24T01:55:33.261 INFO:tasks.rados.rados.0.smithi079.stdout:48: left oid 48 (ObjNum 47 snap 0 seq_num 47) 2024-04-24T01:55:33.261 INFO:tasks.rados.rados.0.smithi079.stdout:49: finishing write tid 3 to smithi07929762-49 2024-04-24T01:55:33.261 INFO:tasks.rados.rados.0.smithi079.stdout:35: finishing write tid 1 to smithi07929762-35 2024-04-24T01:55:33.262 INFO:tasks.rados.rados.0.smithi079.stdout:41: finishing write tid 3 to smithi07929762-41 2024-04-24T01:55:33.262 INFO:tasks.rados.rados.0.smithi079.stdout:41: finishing write tid 4 to smithi07929762-41 2024-04-24T01:55:33.262 INFO:tasks.rados.rados.0.smithi079.stdout:41: oid 41 updating version 0 to 4 2024-04-24T01:55:33.262 INFO:tasks.rados.rados.0.smithi079.stdout:41: oid 41 updating version 4 to 6 2024-04-24T01:55:33.262 INFO:tasks.rados.rados.0.smithi079.stdout:41: oid 41 version 6 is already newer than 6 2024-04-24T01:55:33.262 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 41 v 6 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-04-24T01:55:33.262 INFO:tasks.rados.rados.0.smithi079.stdout:41: left oid 41 (ObjNum 40 snap 0 seq_num 40) 2024-04-24T01:55:33.262 INFO:tasks.rados.rados.0.smithi079.stdout:40: finishing write tid 3 to smithi07929762-40 2024-04-24T01:55:33.262 INFO:tasks.rados.rados.0.smithi079.stdout:41: done (15 left) 2024-04-24T01:55:33.262 INFO:tasks.rados.rados.0.smithi079.stdout:48: done (14 left) 2024-04-24T01:55:33.262 INFO:tasks.rados.rados.0.smithi079.stdout:60: delete oid 33 current snap is 0 2024-04-24T01:55:33.262 INFO:tasks.rados.rados.0.smithi079.stdout:40: finishing write tid 4 to smithi07929762-40 2024-04-24T01:55:33.262 INFO:tasks.rados.rados.0.smithi079.stdout:40: oid 40 updating version 0 to 10 2024-04-24T01:55:33.262 INFO:tasks.rados.rados.0.smithi079.stdout:40: oid 40 updating version 10 to 12 2024-04-24T01:55:33.263 INFO:tasks.rados.rados.0.smithi079.stdout:40: oid 40 version 12 is already newer than 12 2024-04-24T01:55:33.263 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 40 v 12 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-04-24T01:55:33.263 INFO:tasks.rados.rados.0.smithi079.stdout:40: left oid 40 (ObjNum 39 snap 0 seq_num 39) 2024-04-24T01:55:33.263 INFO:tasks.rados.rados.0.smithi079.stdout:51: finishing write tid 2 to smithi07929762-2 2024-04-24T01:55:33.263 INFO:tasks.rados.rados.0.smithi079.stdout:49: finishing write tid 4 to smithi07929762-49 2024-04-24T01:55:33.263 INFO:tasks.rados.rados.0.smithi079.stdout:49: oid 49 updating version 0 to 21 2024-04-24T01:55:33.263 INFO:tasks.rados.rados.0.smithi079.stdout:49: oid 49 version 21 is already newer than 21 2024-04-24T01:55:33.263 INFO:tasks.rados.rados.0.smithi079.stdout:49: oid 49 version 21 is already newer than 19 2024-04-24T01:55:33.263 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 49 v 21 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-04-24T01:55:33.263 INFO:tasks.rados.rados.0.smithi079.stdout:49: left oid 49 (ObjNum 48 snap 0 seq_num 48) 2024-04-24T01:55:33.263 INFO:tasks.rados.rados.0.smithi079.stdout:51: finishing write tid 3 to smithi07929762-2 2024-04-24T01:55:33.263 INFO:tasks.rados.rados.0.smithi079.stdout:43: finishing write tid 2 to smithi07929762-43 2024-04-24T01:55:33.263 INFO:tasks.rados.rados.0.smithi079.stdout:51: finishing write tid 4 to smithi07929762-2 2024-04-24T01:55:33.263 INFO:tasks.rados.rados.0.smithi079.stdout:51: finishing write tid 5 to smithi07929762-2 2024-04-24T01:55:33.264 INFO:tasks.rados.rados.0.smithi079.stdout:51: oid 2 updating version 0 to 7 2024-04-24T01:55:33.264 INFO:tasks.rados.rados.0.smithi079.stdout:51: oid 2 version 7 is already newer than 7 2024-04-24T01:55:33.264 INFO:tasks.rados.rados.0.smithi079.stdout:51: oid 2 version 7 is already newer than 7 2024-04-24T01:55:33.264 INFO:tasks.rados.rados.0.smithi079.stdout:51: oid 2 version 7 is already newer than 4 2024-04-24T01:55:33.264 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 2 v 7 (ObjNum 50 snap 0 seq_num 50) dirty exists 2024-04-24T01:55:33.264 INFO:tasks.rados.rados.0.smithi079.stdout:51: left oid 2 (ObjNum 50 snap 0 seq_num 50) 2024-04-24T01:55:33.265 INFO:tasks.rados.rados.0.smithi079.stdout:53: expect (ObjNum 19 snap 0 seq_num 19) 2024-04-24T01:55:33.376 INFO:tasks.rados.rados.0.smithi079.stdout:43: finishing write tid 3 to smithi07929762-43 2024-04-24T01:55:33.376 INFO:tasks.rados.rados.0.smithi079.stdout:55: finishing write tid 1 to smithi07929762-10 2024-04-24T01:55:33.376 INFO:tasks.rados.rados.0.smithi079.stdout:58: expect (ObjNum 8 snap 0 seq_num 8) 2024-04-24T01:55:33.476 INFO:tasks.rados.rados.0.smithi079.stdout:43: finishing write tid 4 to smithi07929762-43 2024-04-24T01:55:33.477 INFO:tasks.rados.rados.0.smithi079.stdout:43: oid 43 updating version 0 to 6 2024-04-24T01:55:33.477 INFO:tasks.rados.rados.0.smithi079.stdout:43: oid 43 version 6 is already newer than 4 2024-04-24T01:55:33.477 INFO:tasks.rados.rados.0.smithi079.stdout:43: oid 43 version 6 is already newer than 6 2024-04-24T01:55:33.477 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 43 v 6 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-04-24T01:55:33.477 INFO:tasks.rados.rados.0.smithi079.stdout:43: left oid 43 (ObjNum 42 snap 0 seq_num 42) 2024-04-24T01:55:33.477 INFO:tasks.rados.rados.0.smithi079.stdout:50: finishing write tid 1 to smithi07929762-50 2024-04-24T01:55:33.477 INFO:tasks.rados.rados.0.smithi079.stdout:55: finishing write tid 2 to smithi07929762-10 2024-04-24T01:55:33.477 INFO:tasks.rados.rados.0.smithi079.stdout:55: finishing write tid 3 to smithi07929762-10 2024-04-24T01:55:33.477 INFO:tasks.rados.rados.0.smithi079.stdout:52: finishing write tid 1 to smithi07929762-14 2024-04-24T01:55:33.477 INFO:tasks.rados.rados.0.smithi079.stdout:55: finishing write tid 4 to smithi07929762-10 2024-04-24T01:55:33.477 INFO:tasks.rados.rados.0.smithi079.stdout:55: oid 10 updating version 0 to 15 2024-04-24T01:55:33.477 INFO:tasks.rados.rados.0.smithi079.stdout:55: oid 10 version 15 is already newer than 15 2024-04-24T01:55:33.477 INFO:tasks.rados.rados.0.smithi079.stdout:55: oid 10 version 15 is already newer than 15 2024-04-24T01:55:33.478 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 10 v 15 (ObjNum 53 snap 0 seq_num 53) dirty exists 2024-04-24T01:55:33.478 INFO:tasks.rados.rados.0.smithi079.stdout:55: left oid 10 (ObjNum 53 snap 0 seq_num 53) 2024-04-24T01:55:33.478 INFO:tasks.rados.rados.0.smithi079.stdout:47: finishing write tid 2 to smithi07929762-47 2024-04-24T01:55:33.478 INFO:tasks.rados.rados.0.smithi079.stdout:35: finishing write tid 2 to smithi07929762-35 2024-04-24T01:55:33.478 INFO:tasks.rados.rados.0.smithi079.stdout:47: finishing write tid 3 to smithi07929762-47 2024-04-24T01:55:33.478 INFO:tasks.rados.rados.0.smithi079.stdout:35: finishing write tid 3 to smithi07929762-35 2024-04-24T01:55:33.478 INFO:tasks.rados.rados.0.smithi079.stdout:47: finishing write tid 4 to smithi07929762-47 2024-04-24T01:55:33.478 INFO:tasks.rados.rados.0.smithi079.stdout:47: oid 47 updating version 0 to 12 2024-04-24T01:55:33.478 INFO:tasks.rados.rados.0.smithi079.stdout:47: oid 47 version 12 is already newer than 12 2024-04-24T01:55:33.478 INFO:tasks.rados.rados.0.smithi079.stdout:47: oid 47 version 12 is already newer than 10 2024-04-24T01:55:33.478 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 47 v 12 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-04-24T01:55:33.478 INFO:tasks.rados.rados.0.smithi079.stdout:47: left oid 47 (ObjNum 46 snap 0 seq_num 46) 2024-04-24T01:55:33.478 INFO:tasks.rados.rados.0.smithi079.stdout:35: finishing write tid 4 to smithi07929762-35 2024-04-24T01:55:33.478 INFO:tasks.rados.rados.0.smithi079.stdout:35: oid 35 updating version 0 to 9 2024-04-24T01:55:33.478 INFO:tasks.rados.rados.0.smithi079.stdout:35: oid 35 version 9 is already newer than 9 2024-04-24T01:55:33.479 INFO:tasks.rados.rados.0.smithi079.stdout:35: oid 35 version 9 is already newer than 7 2024-04-24T01:55:33.479 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 35 v 9 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-04-24T01:55:33.479 INFO:tasks.rados.rados.0.smithi079.stdout:35: left oid 35 (ObjNum 34 snap 0 seq_num 34) 2024-04-24T01:55:33.479 INFO:tasks.rados.rados.0.smithi079.stdout:50: finishing write tid 2 to smithi07929762-50 2024-04-24T01:55:33.479 INFO:tasks.rados.rados.0.smithi079.stdout:52: finishing write tid 2 to smithi07929762-14 2024-04-24T01:55:33.479 INFO:tasks.rados.rados.0.smithi079.stdout:52: finishing write tid 3 to smithi07929762-14 2024-04-24T01:55:33.479 INFO:tasks.rados.rados.0.smithi079.stdout:50: finishing write tid 3 to smithi07929762-50 2024-04-24T01:55:33.479 INFO:tasks.rados.rados.0.smithi079.stdout:50: finishing write tid 4 to smithi07929762-50 2024-04-24T01:55:33.479 INFO:tasks.rados.rados.0.smithi079.stdout:50: oid 50 updating version 0 to 12 2024-04-24T01:55:33.479 INFO:tasks.rados.rados.0.smithi079.stdout:50: oid 50 version 12 is already newer than 10 2024-04-24T01:55:33.479 INFO:tasks.rados.rados.0.smithi079.stdout:50: oid 50 version 12 is already newer than 12 2024-04-24T01:55:33.479 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 50 v 12 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-04-24T01:55:33.479 INFO:tasks.rados.rados.0.smithi079.stdout:50: left oid 50 (ObjNum 49 snap 0 seq_num 49) 2024-04-24T01:55:33.479 INFO:tasks.rados.rados.0.smithi079.stdout:35: done (14 left) 2024-04-24T01:55:33.479 INFO:tasks.rados.rados.0.smithi079.stdout:40: done (13 left) 2024-04-24T01:55:33.479 INFO:tasks.rados.rados.0.smithi079.stdout:43: done (12 left) 2024-04-24T01:55:33.479 INFO:tasks.rados.rados.0.smithi079.stdout:47: done (11 left) 2024-04-24T01:55:33.480 INFO:tasks.rados.rados.0.smithi079.stdout:49: done (10 left) 2024-04-24T01:55:33.480 INFO:tasks.rados.rados.0.smithi079.stdout:50: done (9 left) 2024-04-24T01:55:33.480 INFO:tasks.rados.rados.0.smithi079.stdout:51: done (8 left) 2024-04-24T01:55:33.480 INFO:tasks.rados.rados.0.smithi079.stdout:53: done (7 left) 2024-04-24T01:55:33.480 INFO:tasks.rados.rados.0.smithi079.stdout:55: done (6 left) 2024-04-24T01:55:33.480 INFO:tasks.rados.rados.0.smithi079.stdout:58: done (5 left) 2024-04-24T01:55:33.480 INFO:tasks.rados.rados.0.smithi079.stdout:60: done (4 left) 2024-04-24T01:55:33.480 INFO:tasks.rados.rados.0.smithi079.stdout:61: delete oid 8 current snap is 0 2024-04-24T01:55:33.480 INFO:tasks.rados.rados.0.smithi079.stdout:52: finishing write tid 4 to smithi07929762-14 2024-04-24T01:55:33.480 INFO:tasks.rados.rados.0.smithi079.stdout:52: oid 14 updating version 0 to 15 2024-04-24T01:55:33.480 INFO:tasks.rados.rados.0.smithi079.stdout:52: oid 14 version 15 is already newer than 15 2024-04-24T01:55:33.480 INFO:tasks.rados.rados.0.smithi079.stdout:52: oid 14 version 15 is already newer than 13 2024-04-24T01:55:33.480 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 14 v 15 (ObjNum 51 snap 0 seq_num 51) dirty exists 2024-04-24T01:55:33.480 INFO:tasks.rados.rados.0.smithi079.stdout:52: left oid 14 (ObjNum 51 snap 0 seq_num 51) 2024-04-24T01:55:33.480 INFO:tasks.rados.rados.0.smithi079.stdout:59: finishing write tid 1 to smithi07929762-24 2024-04-24T01:55:33.480 INFO:tasks.rados.rados.0.smithi079.stdout:56: finishing write tid 1 to smithi07929762-25 2024-04-24T01:55:33.481 INFO:tasks.rados.rados.0.smithi079.stdout:56: finishing write tid 2 to smithi07929762-25 2024-04-24T01:55:33.481 INFO:tasks.rados.rados.0.smithi079.stdout:59: finishing write tid 2 to smithi07929762-24 2024-04-24T01:55:33.481 INFO:tasks.rados.rados.0.smithi079.stdout:56: finishing write tid 3 to smithi07929762-25 2024-04-24T01:55:33.481 INFO:tasks.rados.rados.0.smithi079.stdout:56: finishing write tid 4 to smithi07929762-25 2024-04-24T01:55:33.481 INFO:tasks.rados.rados.0.smithi079.stdout:56: finishing write tid 5 to smithi07929762-25 2024-04-24T01:55:33.481 INFO:tasks.rados.rados.0.smithi079.stdout:56: oid 25 updating version 0 to 16 2024-04-24T01:55:33.481 INFO:tasks.rados.rados.0.smithi079.stdout:56: oid 25 version 16 is already newer than 16 2024-04-24T01:55:33.481 INFO:tasks.rados.rados.0.smithi079.stdout:56: oid 25 version 16 is already newer than 16 2024-04-24T01:55:33.481 INFO:tasks.rados.rados.0.smithi079.stdout:56: oid 25 version 16 is already newer than 16 2024-04-24T01:55:33.481 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 25 v 16 (ObjNum 54 snap 0 seq_num 54) dirty exists 2024-04-24T01:55:33.481 INFO:tasks.rados.rados.0.smithi079.stdout:56: left oid 25 (ObjNum 54 snap 0 seq_num 54) 2024-04-24T01:55:33.481 INFO:tasks.rados.rados.0.smithi079.stdout:59: finishing write tid 3 to smithi07929762-24 2024-04-24T01:55:33.481 INFO:tasks.rados.rados.0.smithi079.stdout:59: finishing write tid 4 to smithi07929762-24 2024-04-24T01:55:33.481 INFO:tasks.rados.rados.0.smithi079.stdout:59: finishing write tid 5 to smithi07929762-24 2024-04-24T01:55:33.481 INFO:tasks.rados.rados.0.smithi079.stdout:59: oid 24 updating version 0 to 13 2024-04-24T01:55:33.482 INFO:tasks.rados.rados.0.smithi079.stdout:59: oid 24 version 13 is already newer than 10 2024-04-24T01:55:33.482 INFO:tasks.rados.rados.0.smithi079.stdout:59: oid 24 version 13 is already newer than 13 2024-04-24T01:55:33.482 INFO:tasks.rados.rados.0.smithi079.stdout:59: oid 24 version 13 is already newer than 13 2024-04-24T01:55:33.482 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 24 v 13 (ObjNum 56 snap 0 seq_num 56) dirty exists 2024-04-24T01:55:33.482 INFO:tasks.rados.rados.0.smithi079.stdout:59: left oid 24 (ObjNum 56 snap 0 seq_num 56) 2024-04-24T01:55:33.482 INFO:tasks.rados.rados.0.smithi079.stdout:57: finishing write tid 1 to smithi07929762-29 2024-04-24T01:55:33.490 INFO:tasks.rados.rados.0.smithi079.stdout:57: finishing write tid 2 to smithi07929762-29 2024-04-24T01:55:33.494 INFO:tasks.rados.rados.0.smithi079.stdout:57: finishing write tid 3 to smithi07929762-29 2024-04-24T01:55:33.495 INFO:tasks.rados.rados.0.smithi079.stdout:57: finishing write tid 4 to smithi07929762-29 2024-04-24T01:55:33.497 INFO:tasks.rados.rados.0.smithi079.stdout:57: finishing write tid 5 to smithi07929762-29 2024-04-24T01:55:33.497 INFO:tasks.rados.rados.0.smithi079.stdout:57: oid 29 updating version 0 to 19 2024-04-24T01:55:33.498 INFO:tasks.rados.rados.0.smithi079.stdout:57: oid 29 version 19 is already newer than 16 2024-04-24T01:55:33.498 INFO:tasks.rados.rados.0.smithi079.stdout:57: oid 29 version 19 is already newer than 19 2024-04-24T01:55:33.498 INFO:tasks.rados.rados.0.smithi079.stdout:57: oid 29 version 19 is already newer than 19 2024-04-24T01:55:33.498 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 29 v 19 (ObjNum 55 snap 0 seq_num 55) dirty exists 2024-04-24T01:55:33.498 INFO:tasks.rados.rados.0.smithi079.stdout:57: left oid 29 (ObjNum 55 snap 0 seq_num 55) 2024-04-24T01:55:33.499 INFO:tasks.rados.rados.0.smithi079.stdout:52: done (4 left) 2024-04-24T01:55:33.499 INFO:tasks.rados.rados.0.smithi079.stdout:56: done (3 left) 2024-04-24T01:55:33.499 INFO:tasks.rados.rados.0.smithi079.stdout:57: done (2 left) 2024-04-24T01:55:33.499 INFO:tasks.rados.rados.0.smithi079.stdout:59: done (1 left) 2024-04-24T01:55:33.499 INFO:tasks.rados.rados.0.smithi079.stdout:61: done (0 left) 2024-04-24T01:55:33.499 INFO:tasks.rados.rados.0.smithi079.stdout:62: delete oid 27 current snap is 0 2024-04-24T01:55:33.514 INFO:tasks.rados.rados.0.smithi079.stdout:62: done (0 left) 2024-04-24T01:55:33.514 INFO:tasks.rados.rados.0.smithi079.stdout:63: read oid 15 snap -1 2024-04-24T01:55:33.515 INFO:tasks.rados.rados.0.smithi079.stdout:63: expect (ObjNum 14 snap 0 seq_num 14) 2024-04-24T01:55:33.515 INFO:tasks.rados.rados.0.smithi079.stdout:64: snap_create 2024-04-24T01:55:33.533 INFO:tasks.rados.rados.0.smithi079.stdout:63: expect (ObjNum 14 snap 0 seq_num 14) 2024-04-24T01:55:33.614 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-04-24T01:55:33.614 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-04-24T01:55:33.614 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-04-24T01:55:33.614 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-04-24T01:55:33.722 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:33.718+0000 7fb01f99d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:33.726 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:33.726+0000 7fb01f99d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:33.742 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T01:55:33.742 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T01:55:33.742 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T01:55:33.742 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T01:55:33.742 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T01:55:33.742 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T01:55:33.742 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T01:55:33.742 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T01:55:33.742 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T01:55:33.743 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T01:55:33.743 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T01:55:33.743 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T01:55:33.744 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T01:55:33.745 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T01:55:33.745 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-04-24T01:55:33.853 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:33.849+0000 7f8eece6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:33.857 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:33.857+0000 7f8eece6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:33.873 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T01:55:33.873 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T01:55:33.873 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T01:55:33.873 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T01:55:33.873 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T01:55:33.873 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T01:55:33.873 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T01:55:33.873 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T01:55:33.873 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T01:55:33.873 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T01:55:33.873 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T01:55:33.873 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T01:55:33.876 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T01:55:33.876 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T01:55:33.877 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-04-24T01:55:33.983 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:33.978+0000 7f5b05aed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:33.986 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:33.987+0000 7f5b05aed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:34.003 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T01:55:34.003 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T01:55:34.003 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T01:55:34.003 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T01:55:34.003 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T01:55:34.003 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T01:55:34.004 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T01:55:34.004 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T01:55:34.004 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T01:55:34.004 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T01:55:34.004 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T01:55:34.004 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T01:55:34.005 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T01:55:34.006 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T01:55:34.006 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-04-24T01:55:34.112 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:34.108+0000 7fc7e079d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:34.116 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:34.116+0000 7fc7e079d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:34.132 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T01:55:34.132 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T01:55:34.132 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T01:55:34.132 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T01:55:34.132 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T01:55:34.132 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T01:55:34.132 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T01:55:34.132 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T01:55:34.132 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T01:55:34.132 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T01:55:34.132 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T01:55:34.133 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T01:55:34.138 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T01:55:34.138 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T01:55:34.212 INFO:tasks.rados.rados.0.smithi079.stdout:63: done (1 left) 2024-04-24T01:55:34.213 INFO:tasks.rados.rados.0.smithi079.stdout:64: done (0 left) 2024-04-24T01:55:34.213 INFO:tasks.rados.rados.0.smithi079.stdout:65: write oid 40 current snap is 1 2024-04-24T01:55:34.213 INFO:tasks.rados.rados.0.smithi079.stdout:65: seq_num 60 ranges {608420=529411,1778096=561011,2881850=126570} 2024-04-24T01:55:34.218 INFO:tasks.rados.rados.0.smithi079.stdout:65: writing smithi07929762-40 from 608420 to 1137831 tid 1 2024-04-24T01:55:34.225 INFO:tasks.rados.rados.0.smithi079.stdout:65: writing smithi07929762-40 from 1778096 to 2339107 tid 2 2024-04-24T01:55:34.230 INFO:tasks.rados.rados.0.smithi079.stdout:65: writing smithi07929762-40 from 2881850 to 3008420 tid 3 2024-04-24T01:55:34.230 INFO:tasks.rados.rados.0.smithi079.stdout:66: read oid 48 snap 0 2024-04-24T01:55:34.230 INFO:tasks.rados.rados.0.smithi079.stdout:66: expect (ObjNum 47 snap 0 seq_num 47) 2024-04-24T01:55:34.230 INFO:tasks.rados.rados.0.smithi079.stdout:67: read oid 28 snap 0 2024-04-24T01:55:34.230 INFO:tasks.rados.rados.0.smithi079.stdout:67: expect (ObjNum 27 snap 0 seq_num 27) 2024-04-24T01:55:34.231 INFO:tasks.rados.rados.0.smithi079.stdout:68: read oid 13 snap -1 2024-04-24T01:55:34.231 INFO:tasks.rados.rados.0.smithi079.stdout:68: expect (ObjNum 12 snap 0 seq_num 12) 2024-04-24T01:55:34.231 INFO:tasks.rados.rados.0.smithi079.stdout:69: read oid 43 snap -1 2024-04-24T01:55:34.231 INFO:tasks.rados.rados.0.smithi079.stdout:69: expect (ObjNum 42 snap 0 seq_num 42) 2024-04-24T01:55:34.231 INFO:tasks.rados.rados.0.smithi079.stdout:70: read oid 36 snap -1 2024-04-24T01:55:34.231 INFO:tasks.rados.rados.0.smithi079.stdout:70: expect (ObjNum 35 snap 0 seq_num 35) 2024-04-24T01:55:34.231 INFO:tasks.rados.rados.0.smithi079.stdout:71: read oid 50 snap -1 2024-04-24T01:55:34.231 INFO:tasks.rados.rados.0.smithi079.stdout:71: expect (ObjNum 49 snap 0 seq_num 49) 2024-04-24T01:55:34.231 INFO:tasks.rados.rados.0.smithi079.stdout:72: read oid 8 snap 0 2024-04-24T01:55:34.231 INFO:tasks.rados.rados.0.smithi079.stdout:72: expect deleted 2024-04-24T01:55:34.231 INFO:tasks.rados.rados.0.smithi079.stdout:73: read oid 27 snap 0 2024-04-24T01:55:34.231 INFO:tasks.rados.rados.0.smithi079.stdout:73: expect deleted 2024-04-24T01:55:34.231 INFO:tasks.rados.rados.0.smithi079.stdout:74: snap_create 2024-04-24T01:55:34.278 INFO:tasks.rados.rados.0.smithi079.stdout:69: expect (ObjNum 42 snap 0 seq_num 42) 2024-04-24T01:55:34.395 INFO:tasks.rados.rados.0.smithi079.stdout:70: expect (ObjNum 35 snap 0 seq_num 35) 2024-04-24T01:55:34.492 INFO:tasks.rados.rados.0.smithi079.stdout:65: finishing write tid 1 to smithi07929762-40 2024-04-24T01:55:34.492 INFO:tasks.rados.rados.0.smithi079.stdout:67: expect (ObjNum 27 snap 0 seq_num 27) 2024-04-24T01:55:34.583 INFO:tasks.rados.rados.0.smithi079.stdout:66: expect (ObjNum 47 snap 0 seq_num 47) 2024-04-24T01:55:34.686 INFO:tasks.rados.rados.0.smithi079.stdout:68: expect (ObjNum 12 snap 0 seq_num 12) 2024-04-24T01:55:34.767 INFO:tasks.rados.rados.0.smithi079.stdout:65: finishing write tid 2 to smithi07929762-40 2024-04-24T01:55:34.767 INFO:tasks.rados.rados.0.smithi079.stdout:71: expect (ObjNum 49 snap 0 seq_num 49) 2024-04-24T01:55:34.872 INFO:tasks.rados.rados.0.smithi079.stdout:65: finishing write tid 3 to smithi07929762-40 2024-04-24T01:55:34.873 INFO:tasks.rados.rados.0.smithi079.stdout:65: finishing write tid 4 to smithi07929762-40 2024-04-24T01:55:34.873 INFO:tasks.rados.rados.0.smithi079.stdout:65: finishing write tid 5 to smithi07929762-40 2024-04-24T01:55:34.873 INFO:tasks.rados.rados.0.smithi079.stdout:65: oid 40 updating version 0 to 18 2024-04-24T01:55:34.873 INFO:tasks.rados.rados.0.smithi079.stdout:65: oid 40 version 18 is already newer than 18 2024-04-24T01:55:34.873 INFO:tasks.rados.rados.0.smithi079.stdout:65: oid 40 version 18 is already newer than 18 2024-04-24T01:55:34.873 INFO:tasks.rados.rados.0.smithi079.stdout:65: oid 40 version 18 is already newer than 18 2024-04-24T01:55:34.873 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 40 v 18 (ObjNum 60 snap 1 seq_num 60) dirty exists 2024-04-24T01:55:34.873 INFO:tasks.rados.rados.0.smithi079.stdout:65: left oid 40 (ObjNum 60 snap 1 seq_num 60) 2024-04-24T01:55:35.215 INFO:tasks.rados.rados.0.smithi079.stdout:65: done (9 left) 2024-04-24T01:55:35.215 INFO:tasks.rados.rados.0.smithi079.stdout:66: done (8 left) 2024-04-24T01:55:35.216 INFO:tasks.rados.rados.0.smithi079.stdout:67: done (7 left) 2024-04-24T01:55:35.216 INFO:tasks.rados.rados.0.smithi079.stdout:68: done (6 left) 2024-04-24T01:55:35.216 INFO:tasks.rados.rados.0.smithi079.stdout:69: done (5 left) 2024-04-24T01:55:35.216 INFO:tasks.rados.rados.0.smithi079.stdout:70: done (4 left) 2024-04-24T01:55:35.216 INFO:tasks.rados.rados.0.smithi079.stdout:71: done (3 left) 2024-04-24T01:55:35.216 INFO:tasks.rados.rados.0.smithi079.stdout:72: done (2 left) 2024-04-24T01:55:35.216 INFO:tasks.rados.rados.0.smithi079.stdout:73: done (1 left) 2024-04-24T01:55:35.216 INFO:tasks.rados.rados.0.smithi079.stdout:74: done (0 left) 2024-04-24T01:55:35.216 INFO:tasks.rados.rados.0.smithi079.stdout:75: write oid 45 current snap is 2 2024-04-24T01:55:35.216 INFO:tasks.rados.rados.0.smithi079.stdout:75: seq_num 61 ranges {625227=721013,1952931=704669} 2024-04-24T01:55:35.222 INFO:tasks.rados.rados.0.smithi079.stdout:75: writing smithi07929762-45 from 625227 to 1346240 tid 1 2024-04-24T01:55:35.229 INFO:tasks.rados.rados.0.smithi079.stdout:75: writing smithi07929762-45 from 1952931 to 2657600 tid 2 2024-04-24T01:55:35.229 INFO:tasks.rados.rados.0.smithi079.stdout:76: read oid 16 snap -1 2024-04-24T01:55:35.230 INFO:tasks.rados.rados.0.smithi079.stdout:76: expect (ObjNum 15 snap 0 seq_num 15) 2024-04-24T01:55:35.230 INFO:tasks.rados.rados.0.smithi079.stdout:77: write oid 46 current snap is 2 2024-04-24T01:55:35.230 INFO:tasks.rados.rados.0.smithi079.stdout:77: seq_num 62 ranges {642034=796262,1895060=564680} 2024-04-24T01:55:35.238 INFO:tasks.rados.rados.0.smithi079.stdout:77: writing smithi07929762-46 from 642034 to 1438296 tid 1 2024-04-24T01:55:35.244 INFO:tasks.rados.rados.0.smithi079.stdout:77: writing smithi07929762-46 from 1895060 to 2459740 tid 2 2024-04-24T01:55:35.244 INFO:tasks.rados.rados.0.smithi079.stdout:78: read oid 26 snap 0 2024-04-24T01:55:35.244 INFO:tasks.rados.rados.0.smithi079.stdout:78: expect (ObjNum 25 snap 0 seq_num 25) 2024-04-24T01:55:35.244 INFO:tasks.rados.rados.0.smithi079.stdout:79: read oid 7 snap -1 2024-04-24T01:55:35.244 INFO:tasks.rados.rados.0.smithi079.stdout:79: expect (ObjNum 6 snap 0 seq_num 6) 2024-04-24T01:55:35.244 INFO:tasks.rados.rados.0.smithi079.stdout:80: write oid 44 current snap is 2 2024-04-24T01:55:35.244 INFO:tasks.rados.rados.0.smithi079.stdout:80: seq_num 63 ranges {658841=471511,1553542=708338,2964119=94722} 2024-04-24T01:55:35.250 INFO:tasks.rados.rados.0.smithi079.stdout:80: writing smithi07929762-44 from 658841 to 1130352 tid 1 2024-04-24T01:55:35.257 INFO:tasks.rados.rados.0.smithi079.stdout:80: writing smithi07929762-44 from 1553542 to 2261880 tid 2 2024-04-24T01:55:35.262 INFO:tasks.rados.rados.0.smithi079.stdout:80: writing smithi07929762-44 from 2964119 to 3058841 tid 3 2024-04-24T01:55:35.262 INFO:tasks.rados.rados.0.smithi079.stdout:81: read oid 30 snap -1 2024-04-24T01:55:35.262 INFO:tasks.rados.rados.0.smithi079.stdout:81: expect (ObjNum 29 snap 0 seq_num 29) 2024-04-24T01:55:35.262 INFO:tasks.rados.rados.0.smithi079.stdout:82: snap_remove snap 1 2024-04-24T01:55:35.266 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T01:55:35.397 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:35.396+0000 7f07034ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:35.401 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:35.401+0000 7f07034ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:36.218 INFO:tasks.rados.rados.0.smithi079.stdout:82: done (7 left) 2024-04-24T01:55:36.218 INFO:tasks.rados.rados.0.smithi079.stdout:83: delete oid 49 current snap is 2 2024-04-24T01:55:36.218 INFO:tasks.rados.rados.0.smithi079.stdout:75: finishing write tid 1 to smithi07929762-45 2024-04-24T01:55:36.219 INFO:tasks.rados.rados.0.smithi079.stdout:78: expect (ObjNum 25 snap 0 seq_num 25) 2024-04-24T01:55:36.321 INFO:tasks.rados.rados.0.smithi079.stdout:79: expect (ObjNum 6 snap 0 seq_num 6) 2024-04-24T01:55:36.401 INFO:tasks.rados.rados.0.smithi079.stdout:75: finishing write tid 2 to smithi07929762-45 2024-04-24T01:55:36.401 INFO:tasks.rados.rados.0.smithi079.stdout:75: finishing write tid 3 to smithi07929762-45 2024-04-24T01:55:36.401 INFO:tasks.rados.rados.0.smithi079.stdout:76: expect (ObjNum 15 snap 0 seq_num 15) 2024-04-24T01:55:36.486 INFO:tasks.rados.rados.0.smithi079.stdout:75: finishing write tid 4 to smithi07929762-45 2024-04-24T01:55:36.486 INFO:tasks.rados.rados.0.smithi079.stdout:75: oid 45 updating version 0 to 19 2024-04-24T01:55:36.486 INFO:tasks.rados.rados.0.smithi079.stdout:75: oid 45 version 19 is already newer than 19 2024-04-24T01:55:36.486 INFO:tasks.rados.rados.0.smithi079.stdout:75: oid 45 version 19 is already newer than 19 2024-04-24T01:55:36.486 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 45 v 19 (ObjNum 61 snap 2 seq_num 61) dirty exists 2024-04-24T01:55:36.486 INFO:tasks.rados.rados.0.smithi079.stdout:75: left oid 45 (ObjNum 61 snap 2 seq_num 61) 2024-04-24T01:55:36.486 INFO:tasks.rados.rados.0.smithi079.stdout:75: done (7 left) 2024-04-24T01:55:36.487 INFO:tasks.rados.rados.0.smithi079.stdout:76: done (6 left) 2024-04-24T01:55:36.487 INFO:tasks.rados.rados.0.smithi079.stdout:78: done (5 left) 2024-04-24T01:55:36.487 INFO:tasks.rados.rados.0.smithi079.stdout:79: done (4 left) 2024-04-24T01:55:36.487 INFO:tasks.rados.rados.0.smithi079.stdout:83: done (3 left) 2024-04-24T01:55:36.487 INFO:tasks.rados.rados.0.smithi079.stdout:84: read oid 26 snap -1 2024-04-24T01:55:36.487 INFO:tasks.rados.rados.0.smithi079.stdout:84: expect (ObjNum 25 snap 0 seq_num 25) 2024-04-24T01:55:36.487 INFO:tasks.rados.rados.0.smithi079.stdout:85: read oid 28 snap 0 2024-04-24T01:55:36.487 INFO:tasks.rados.rados.0.smithi079.stdout:85: expect (ObjNum 27 snap 0 seq_num 27) 2024-04-24T01:55:36.487 INFO:tasks.rados.rados.0.smithi079.stdout:86: write oid 3 current snap is 2 2024-04-24T01:55:36.487 INFO:tasks.rados.rados.0.smithi079.stdout:80: finishing write tid 1 to smithi07929762-44 2024-04-24T01:55:36.487 INFO:tasks.rados.rados.0.smithi079.stdout:86: seq_num 65 ranges {692455=622009,2070506=712007} 2024-04-24T01:55:36.493 INFO:tasks.rados.rados.0.smithi079.stdout:86: writing smithi07929762-3 from 692455 to 1314464 tid 1 2024-04-24T01:55:36.504 INFO:tasks.rados.rados.0.smithi079.stdout:86: writing smithi07929762-3 from 2070506 to 2782513 tid 2 2024-04-24T01:55:36.504 INFO:tasks.rados.rados.0.smithi079.stdout:87: snap_create 2024-04-24T01:55:36.504 INFO:tasks.rados.rados.0.smithi079.stdout:77: finishing write tid 1 to smithi07929762-46 2024-04-24T01:55:36.504 INFO:tasks.rados.rados.0.smithi079.stdout:77: finishing write tid 2 to smithi07929762-46 2024-04-24T01:55:36.504 INFO:tasks.rados.rados.0.smithi079.stdout:80: finishing write tid 2 to smithi07929762-44 2024-04-24T01:55:36.504 INFO:tasks.rados.rados.0.smithi079.stdout:77: finishing write tid 3 to smithi07929762-46 2024-04-24T01:55:36.505 INFO:tasks.rados.rados.0.smithi079.stdout:80: finishing write tid 3 to smithi07929762-44 2024-04-24T01:55:36.505 INFO:tasks.rados.rados.0.smithi079.stdout:77: finishing write tid 4 to smithi07929762-46 2024-04-24T01:55:36.505 INFO:tasks.rados.rados.0.smithi079.stdout:77: oid 46 updating version 0 to 16 2024-04-24T01:55:36.505 INFO:tasks.rados.rados.0.smithi079.stdout:77: oid 46 version 16 is already newer than 16 2024-04-24T01:55:36.505 INFO:tasks.rados.rados.0.smithi079.stdout:77: oid 46 version 16 is already newer than 16 2024-04-24T01:55:36.505 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 46 v 16 (ObjNum 62 snap 2 seq_num 62) dirty exists 2024-04-24T01:55:36.505 INFO:tasks.rados.rados.0.smithi079.stdout:77: left oid 46 (ObjNum 62 snap 2 seq_num 62) 2024-04-24T01:55:36.505 INFO:tasks.rados.rados.0.smithi079.stdout:81: expect (ObjNum 29 snap 0 seq_num 29) 2024-04-24T01:55:36.619 INFO:tasks.rados.rados.0.smithi079.stdout:80: finishing write tid 4 to smithi07929762-44 2024-04-24T01:55:36.619 INFO:tasks.rados.rados.0.smithi079.stdout:80: finishing write tid 5 to smithi07929762-44 2024-04-24T01:55:36.619 INFO:tasks.rados.rados.0.smithi079.stdout:80: oid 44 updating version 0 to 14 2024-04-24T01:55:36.619 INFO:tasks.rados.rados.0.smithi079.stdout:80: oid 44 version 14 is already newer than 14 2024-04-24T01:55:36.619 INFO:tasks.rados.rados.0.smithi079.stdout:80: oid 44 version 14 is already newer than 14 2024-04-24T01:55:36.619 INFO:tasks.rados.rados.0.smithi079.stdout:80: oid 44 version 14 is already newer than 14 2024-04-24T01:55:36.619 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 44 v 14 (ObjNum 63 snap 2 seq_num 63) dirty exists 2024-04-24T01:55:36.619 INFO:tasks.rados.rados.0.smithi079.stdout:80: left oid 44 (ObjNum 63 snap 2 seq_num 63) 2024-04-24T01:55:36.619 INFO:tasks.rados.rados.0.smithi079.stdout:86: finishing write tid 1 to smithi07929762-3 2024-04-24T01:55:36.619 INFO:tasks.rados.rados.0.smithi079.stdout:86: finishing write tid 2 to smithi07929762-3 2024-04-24T01:55:36.620 INFO:tasks.rados.rados.0.smithi079.stdout:84: expect (ObjNum 25 snap 0 seq_num 25) 2024-04-24T01:55:36.709 INFO:tasks.rados.rados.0.smithi079.stdout:86: finishing write tid 3 to smithi07929762-3 2024-04-24T01:55:36.709 INFO:tasks.rados.rados.0.smithi079.stdout:86: finishing write tid 4 to smithi07929762-3 2024-04-24T01:55:36.709 INFO:tasks.rados.rados.0.smithi079.stdout:86: oid 3 updating version 0 to 27 2024-04-24T01:55:36.709 INFO:tasks.rados.rados.0.smithi079.stdout:86: oid 3 version 27 is already newer than 27 2024-04-24T01:55:36.709 INFO:tasks.rados.rados.0.smithi079.stdout:86: oid 3 version 27 is already newer than 27 2024-04-24T01:55:36.710 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 3 v 27 (ObjNum 65 snap 2 seq_num 65) dirty exists 2024-04-24T01:55:36.710 INFO:tasks.rados.rados.0.smithi079.stdout:86: left oid 3 (ObjNum 65 snap 2 seq_num 65) 2024-04-24T01:55:36.710 INFO:tasks.rados.rados.0.smithi079.stdout:85: expect (ObjNum 27 snap 0 seq_num 27) 2024-04-24T01:55:37.221 INFO:tasks.rados.rados.0.smithi079.stdout:77: done (6 left) 2024-04-24T01:55:37.222 INFO:tasks.rados.rados.0.smithi079.stdout:80: done (5 left) 2024-04-24T01:55:37.222 INFO:tasks.rados.rados.0.smithi079.stdout:81: done (4 left) 2024-04-24T01:55:37.222 INFO:tasks.rados.rados.0.smithi079.stdout:84: done (3 left) 2024-04-24T01:55:37.222 INFO:tasks.rados.rados.0.smithi079.stdout:85: done (2 left) 2024-04-24T01:55:37.222 INFO:tasks.rados.rados.0.smithi079.stdout:86: done (1 left) 2024-04-24T01:55:37.222 INFO:tasks.rados.rados.0.smithi079.stdout:87: done (0 left) 2024-04-24T01:55:37.222 INFO:tasks.rados.rados.0.smithi079.stdout:88: write oid 6 current snap is 3 2024-04-24T01:55:37.222 INFO:tasks.rados.rados.0.smithi079.stdout:88: seq_num 66 ranges {709262=697258,2012635=572018,3046388=62874} 2024-04-24T01:55:37.228 INFO:tasks.rados.rados.0.smithi079.stdout:88: writing smithi07929762-6 from 709262 to 1406520 tid 1 2024-04-24T01:55:37.235 INFO:tasks.rados.rados.0.smithi079.stdout:88: writing smithi07929762-6 from 2012635 to 2584653 tid 2 2024-04-24T01:55:37.238 INFO:tasks.rados.rados.0.smithi079.stdout:88: writing smithi07929762-6 from 3046388 to 3109262 tid 3 2024-04-24T01:55:37.239 INFO:tasks.rados.rados.0.smithi079.stdout:89: write oid 7 current snap is 3 2024-04-24T01:55:37.239 INFO:tasks.rados.rados.0.smithi079.stdout:89: seq_num 67 ranges {726069=772507,2071117=715676} 2024-04-24T01:55:37.246 INFO:tasks.rados.rados.0.smithi079.stdout:89: writing smithi07929762-7 from 726069 to 1498576 tid 1 2024-04-24T01:55:37.254 INFO:tasks.rados.rados.0.smithi079.stdout:89: writing smithi07929762-7 from 2071117 to 2786793 tid 2 2024-04-24T01:55:37.254 INFO:tasks.rados.rados.0.smithi079.stdout:90: write oid 22 current snap is 3 2024-04-24T01:55:37.254 INFO:tasks.rados.rados.0.smithi079.stdout:90: seq_num 68 ranges {742876=447756,1729599=575687,3101234=41642} 2024-04-24T01:55:37.260 INFO:tasks.rados.rados.0.smithi079.stdout:90: writing smithi07929762-22 from 742876 to 1190632 tid 1 2024-04-24T01:55:37.266 INFO:tasks.rados.rados.0.smithi079.stdout:90: writing smithi07929762-22 from 1729599 to 2305286 tid 2 2024-04-24T01:55:37.270 INFO:tasks.rados.rados.0.smithi079.stdout:90: writing smithi07929762-22 from 3101234 to 3142876 tid 3 2024-04-24T01:55:37.271 INFO:tasks.rados.rados.0.smithi079.stdout:91: delete oid 14 current snap is 3 2024-04-24T01:55:37.324 INFO:tasks.rados.rados.0.smithi079.stdout:90: finishing write tid 1 to smithi07929762-22 2024-04-24T01:55:37.353 INFO:tasks.rados.rados.0.smithi079.stdout:88: finishing write tid 1 to smithi07929762-6 2024-04-24T01:55:37.353 INFO:tasks.rados.rados.0.smithi079.stdout:89: finishing write tid 1 to smithi07929762-7 2024-04-24T01:55:37.357 INFO:tasks.rados.rados.0.smithi079.stdout:90: finishing write tid 2 to smithi07929762-22 2024-04-24T01:55:37.362 INFO:tasks.rados.rados.0.smithi079.stdout:89: finishing write tid 2 to smithi07929762-7 2024-04-24T01:55:37.366 INFO:tasks.rados.rados.0.smithi079.stdout:88: finishing write tid 2 to smithi07929762-6 2024-04-24T01:55:37.374 INFO:tasks.rados.rados.0.smithi079.stdout:90: finishing write tid 3 to smithi07929762-22 2024-04-24T01:55:37.374 INFO:tasks.rados.rados.0.smithi079.stdout:90: finishing write tid 4 to smithi07929762-22 2024-04-24T01:55:37.374 INFO:tasks.rados.rados.0.smithi079.stdout:90: finishing write tid 5 to smithi07929762-22 2024-04-24T01:55:37.375 INFO:tasks.rados.rados.0.smithi079.stdout:90: oid 22 updating version 0 to 11 2024-04-24T01:55:37.375 INFO:tasks.rados.rados.0.smithi079.stdout:90: oid 22 version 11 is already newer than 11 2024-04-24T01:55:37.375 INFO:tasks.rados.rados.0.smithi079.stdout:90: oid 22 version 11 is already newer than 11 2024-04-24T01:55:37.375 INFO:tasks.rados.rados.0.smithi079.stdout:90: oid 22 version 11 is already newer than 11 2024-04-24T01:55:37.375 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 22 v 11 (ObjNum 68 snap 3 seq_num 68) dirty exists 2024-04-24T01:55:37.375 INFO:tasks.rados.rados.0.smithi079.stdout:90: left oid 22 (ObjNum 68 snap 3 seq_num 68) 2024-04-24T01:55:37.377 INFO:tasks.rados.rados.0.smithi079.stdout:89: finishing write tid 3 to smithi07929762-7 2024-04-24T01:55:37.378 INFO:tasks.rados.rados.0.smithi079.stdout:88: finishing write tid 3 to smithi07929762-6 2024-04-24T01:55:37.379 INFO:tasks.rados.rados.0.smithi079.stdout:89: finishing write tid 4 to smithi07929762-7 2024-04-24T01:55:37.379 INFO:tasks.rados.rados.0.smithi079.stdout:89: oid 7 updating version 0 to 20 2024-04-24T01:55:37.379 INFO:tasks.rados.rados.0.smithi079.stdout:89: oid 7 version 20 is already newer than 20 2024-04-24T01:55:37.379 INFO:tasks.rados.rados.0.smithi079.stdout:89: oid 7 version 20 is already newer than 20 2024-04-24T01:55:37.379 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 7 v 20 (ObjNum 67 snap 3 seq_num 67) dirty exists 2024-04-24T01:55:37.379 INFO:tasks.rados.rados.0.smithi079.stdout:89: left oid 7 (ObjNum 67 snap 3 seq_num 67) 2024-04-24T01:55:37.383 INFO:tasks.rados.rados.0.smithi079.stdout:89: done (3 left) 2024-04-24T01:55:37.383 INFO:tasks.rados.rados.0.smithi079.stdout:90: done (2 left) 2024-04-24T01:55:37.383 INFO:tasks.rados.rados.0.smithi079.stdout:91: done (1 left) 2024-04-24T01:55:37.383 INFO:tasks.rados.rados.0.smithi079.stdout:92: write (excl) oid 3 current snap is 3 2024-04-24T01:55:37.383 INFO:tasks.rados.rados.0.smithi079.stdout:92: seq_num 70 ranges {776490=714607,2246563=579356} 2024-04-24T01:55:37.391 INFO:tasks.rados.rados.0.smithi079.stdout:92: writing smithi07929762-3 from 776490 to 1491097 tid 1 2024-04-24T01:55:37.404 INFO:tasks.rados.rados.0.smithi079.stdout:92: writing smithi07929762-3 from 2246563 to 2825919 tid 2 2024-04-24T01:55:37.404 INFO:tasks.rados.rados.0.smithi079.stdout:93: write oid 28 current snap is 3 2024-04-24T01:55:37.404 INFO:tasks.rados.rados.0.smithi079.stdout:93: seq_num 71 ranges {793297=789856,2305045=723014} 2024-04-24T01:55:37.407 INFO:tasks.rados.rados.0.smithi079.stdout:93: writing smithi07929762-28 from 793297 to 1583153 tid 1 2024-04-24T01:55:37.418 INFO:tasks.rados.rados.0.smithi079.stdout:93: writing smithi07929762-28 from 2305045 to 3028059 tid 2 2024-04-24T01:55:37.419 INFO:tasks.rados.rados.0.smithi079.stdout:94: read oid 30 snap -1 2024-04-24T01:55:37.419 INFO:tasks.rados.rados.0.smithi079.stdout:94: expect (ObjNum 29 snap 0 seq_num 29) 2024-04-24T01:55:37.419 INFO:tasks.rados.rados.0.smithi079.stdout:95: read oid 36 snap -1 2024-04-24T01:55:37.419 INFO:tasks.rados.rados.0.smithi079.stdout:95: expect (ObjNum 35 snap 0 seq_num 35) 2024-04-24T01:55:37.419 INFO:tasks.rados.rados.0.smithi079.stdout:96: delete oid 5 current snap is 3 2024-04-24T01:55:37.420 INFO:tasks.rados.rados.0.smithi079.stdout:88: finishing write tid 4 to smithi07929762-6 2024-04-24T01:55:37.420 INFO:tasks.rados.rados.0.smithi079.stdout:88: finishing write tid 5 to smithi07929762-6 2024-04-24T01:55:37.420 INFO:tasks.rados.rados.0.smithi079.stdout:88: oid 6 updating version 0 to 20 2024-04-24T01:55:37.420 INFO:tasks.rados.rados.0.smithi079.stdout:88: oid 6 version 20 is already newer than 20 2024-04-24T01:55:37.420 INFO:tasks.rados.rados.0.smithi079.stdout:88: oid 6 version 20 is already newer than 20 2024-04-24T01:55:37.420 INFO:tasks.rados.rados.0.smithi079.stdout:88: oid 6 version 20 is already newer than 20 2024-04-24T01:55:37.421 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 6 v 20 (ObjNum 66 snap 3 seq_num 66) dirty exists 2024-04-24T01:55:37.421 INFO:tasks.rados.rados.0.smithi079.stdout:88: left oid 6 (ObjNum 66 snap 3 seq_num 66) 2024-04-24T01:55:37.459 INFO:tasks.rados.rados.0.smithi079.stdout:93: finishing write tid 1 to smithi07929762-28 2024-04-24T01:55:37.478 INFO:tasks.rados.rados.0.smithi079.stdout:92: finishing write tid 1 to smithi07929762-3 2024-04-24T01:55:37.492 INFO:tasks.rados.rados.0.smithi079.stdout:93: finishing write tid 2 to smithi07929762-28 2024-04-24T01:55:37.498 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T01:55:37.499 INFO:tasks.rados.rados.0.smithi079.stdout:92: finishing write tid 2 to smithi07929762-3 2024-04-24T01:55:37.500 INFO:tasks.rados.rados.0.smithi079.stdout:95: expect (ObjNum 35 snap 0 seq_num 35) 2024-04-24T01:55:37.601 INFO:tasks.rados.rados.0.smithi079.stdout:94: expect (ObjNum 29 snap 0 seq_num 29) 2024-04-24T01:55:37.613 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:37.609+0000 7f1d2fb5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:37.614 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:37.614+0000 7f1d2fb5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:37.703 INFO:tasks.rados.rados.0.smithi079.stdout:93: finishing write tid 3 to smithi07929762-28 2024-04-24T01:55:37.703 INFO:tasks.rados.rados.0.smithi079.stdout:92: finishing write tid 3 to smithi07929762-3 2024-04-24T01:55:37.703 INFO:tasks.rados.rados.0.smithi079.stdout:93: finishing write tid 4 to smithi07929762-28 2024-04-24T01:55:37.703 INFO:tasks.rados.rados.0.smithi079.stdout:93: oid 28 updating version 0 to 25 2024-04-24T01:55:37.703 INFO:tasks.rados.rados.0.smithi079.stdout:93: oid 28 version 25 is already newer than 25 2024-04-24T01:55:37.704 INFO:tasks.rados.rados.0.smithi079.stdout:93: oid 28 version 25 is already newer than 25 2024-04-24T01:55:37.704 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 28 v 25 (ObjNum 71 snap 3 seq_num 71) dirty exists 2024-04-24T01:55:37.704 INFO:tasks.rados.rados.0.smithi079.stdout:93: left oid 28 (ObjNum 71 snap 3 seq_num 71) 2024-04-24T01:55:37.704 INFO:tasks.rados.rados.0.smithi079.stdout:92: finishing write tid 4 to smithi07929762-3 2024-04-24T01:55:37.704 INFO:tasks.rados.rados.0.smithi079.stdout:92: oid 3 updating version 0 to 31 2024-04-24T01:55:37.704 INFO:tasks.rados.rados.0.smithi079.stdout:92: oid 3 version 31 is already newer than 29 2024-04-24T01:55:37.704 INFO:tasks.rados.rados.0.smithi079.stdout:92: oid 3 version 31 is already newer than 31 2024-04-24T01:55:37.704 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 3 v 31 (ObjNum 70 snap 3 seq_num 70) dirty exists 2024-04-24T01:55:37.704 INFO:tasks.rados.rados.0.smithi079.stdout:92: left oid 3 (ObjNum 70 snap 3 seq_num 70) 2024-04-24T01:55:37.704 INFO:tasks.rados.rados.0.smithi079.stdout:88: done (5 left) 2024-04-24T01:55:37.704 INFO:tasks.rados.rados.0.smithi079.stdout:92: done (4 left) 2024-04-24T01:55:37.704 INFO:tasks.rados.rados.0.smithi079.stdout:93: done (3 left) 2024-04-24T01:55:37.704 INFO:tasks.rados.rados.0.smithi079.stdout:94: done (2 left) 2024-04-24T01:55:37.705 INFO:tasks.rados.rados.0.smithi079.stdout:95: done (1 left) 2024-04-24T01:55:37.705 INFO:tasks.rados.rados.0.smithi079.stdout:96: done (0 left) 2024-04-24T01:55:37.705 INFO:tasks.rados.rados.0.smithi079.stdout:97: snap_remove snap 0 2024-04-24T01:55:38.224 INFO:tasks.rados.rados.0.smithi079.stdout:97: done (0 left) 2024-04-24T01:55:38.225 INFO:tasks.rados.rados.0.smithi079.stdout:98: write (excl) oid 43 current snap is 3 2024-04-24T01:55:38.225 INFO:tasks.rados.rados.0.smithi079.stdout:98: seq_num 73 ranges {426911=540354,1622009=726683,3121996=104915} 2024-04-24T01:55:38.229 INFO:tasks.rados.rados.0.smithi079.stdout:98: writing smithi07929762-43 from 426911 to 967265 tid 1 2024-04-24T01:55:38.237 INFO:tasks.rados.rados.0.smithi079.stdout:98: writing smithi07929762-43 from 1622009 to 2348692 tid 2 2024-04-24T01:55:38.242 INFO:tasks.rados.rados.0.smithi079.stdout:98: writing smithi07929762-43 from 3121996 to 3226911 tid 3 2024-04-24T01:55:38.242 INFO:tasks.rados.rados.0.smithi079.stdout:99: snap_remove snap 2 2024-04-24T01:55:39.139 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-04-24T01:55:39.140 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-04-24T01:55:39.230 INFO:tasks.rados.rados.0.smithi079.stdout:99: done (1 left) 2024-04-24T01:55:39.231 INFO:tasks.rados.rados.0.smithi079.stdout:100: write oid 27 current snap is 3 2024-04-24T01:55:39.231 INFO:tasks.rados.rados.0.smithi079.stdout:100: seq_num 74 ranges {443718=615603,1564138=470341,2633066=610652} 2024-04-24T01:55:39.236 INFO:tasks.rados.rados.0.smithi079.stdout:100: writing smithi07929762-27 from 443718 to 1059321 tid 1 2024-04-24T01:55:39.242 INFO:tasks.rados.rados.0.smithi079.stdout:100: writing smithi07929762-27 from 1564138 to 2034479 tid 2 2024-04-24T01:55:39.250 INFO:tasks.rados.rados.0.smithi079.stdout:100: writing smithi07929762-27 from 2633066 to 3243718 tid 3 2024-04-24T01:55:39.250 INFO:tasks.rados.rados.0.smithi079.stdout:101: read oid 34 snap -1 2024-04-24T01:55:39.250 INFO:tasks.rados.rados.0.smithi079.stdout:101: expect (ObjNum 33 snap 0 seq_num 33) 2024-04-24T01:55:39.250 INFO:tasks.rados.rados.0.smithi079.stdout:102: read oid 4 snap -1 2024-04-24T01:55:39.250 INFO:tasks.rados.rados.0.smithi079.stdout:102: expect (ObjNum 3 snap 0 seq_num 3) 2024-04-24T01:55:39.250 INFO:tasks.rados.rados.0.smithi079.stdout:103: read oid 19 snap -1 2024-04-24T01:55:39.250 INFO:tasks.rados.rados.0.smithi079.stdout:103: expect (ObjNum 18 snap 0 seq_num 18) 2024-04-24T01:55:39.251 INFO:tasks.rados.rados.0.smithi079.stdout:104: read oid 35 snap -1 2024-04-24T01:55:39.251 INFO:tasks.rados.rados.0.smithi079.stdout:104: expect (ObjNum 34 snap 0 seq_num 34) 2024-04-24T01:55:39.251 INFO:tasks.rados.rados.0.smithi079.stdout:105: read oid 45 snap -1 2024-04-24T01:55:39.251 INFO:tasks.rados.rados.0.smithi079.stdout:105: expect (ObjNum 61 snap 2 seq_num 61) 2024-04-24T01:55:39.251 INFO:tasks.rados.rados.0.smithi079.stdout:106: read oid 26 snap -1 2024-04-24T01:55:39.251 INFO:tasks.rados.rados.0.smithi079.stdout:106: expect (ObjNum 25 snap 0 seq_num 25) 2024-04-24T01:55:39.251 INFO:tasks.rados.rados.0.smithi079.stdout:107: delete oid 15 current snap is 3 2024-04-24T01:55:39.251 INFO:tasks.rados.rados.0.smithi079.stdout:98: finishing write tid 1 to smithi07929762-43 2024-04-24T01:55:39.251 INFO:tasks.rados.rados.0.smithi079.stdout:98: finishing write tid 2 to smithi07929762-43 2024-04-24T01:55:39.251 INFO:tasks.rados.rados.0.smithi079.stdout:98: finishing write tid 3 to smithi07929762-43 2024-04-24T01:55:39.251 INFO:tasks.rados.rados.0.smithi079.stdout:98: finishing write tid 4 to smithi07929762-43 2024-04-24T01:55:39.251 INFO:tasks.rados.rados.0.smithi079.stdout:98: finishing write tid 5 to smithi07929762-43 2024-04-24T01:55:39.251 INFO:tasks.rados.rados.0.smithi079.stdout:98: oid 43 updating version 0 to 11 2024-04-24T01:55:39.251 INFO:tasks.rados.rados.0.smithi079.stdout:98: oid 43 version 11 is already newer than 11 2024-04-24T01:55:39.251 INFO:tasks.rados.rados.0.smithi079.stdout:98: oid 43 version 11 is already newer than 11 2024-04-24T01:55:39.252 INFO:tasks.rados.rados.0.smithi079.stdout:98: oid 43 version 11 is already newer than 8 2024-04-24T01:55:39.252 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 43 v 11 (ObjNum 73 snap 3 seq_num 73) dirty exists 2024-04-24T01:55:39.252 INFO:tasks.rados.rados.0.smithi079.stdout:98: left oid 43 (ObjNum 73 snap 3 seq_num 73) 2024-04-24T01:55:39.264 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:39.255+0000 7f174055d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:39.271 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:39.270+0000 7f174055d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:39.343 INFO:tasks.rados.rados.0.smithi079.stdout:106: expect (ObjNum 25 snap 0 seq_num 25) 2024-04-24T01:55:39.447 INFO:tasks.rados.rados.0.smithi079.stdout:98: done (8 left) 2024-04-24T01:55:39.448 INFO:tasks.rados.rados.0.smithi079.stdout:106: done (7 left) 2024-04-24T01:55:39.448 INFO:tasks.rados.rados.0.smithi079.stdout:107: done (6 left) 2024-04-24T01:55:39.448 INFO:tasks.rados.rados.0.smithi079.stdout:108: snap_create 2024-04-24T01:55:39.448 INFO:tasks.rados.rados.0.smithi079.stdout:102: expect (ObjNum 3 snap 0 seq_num 3) 2024-04-24T01:55:39.524 INFO:tasks.rados.rados.0.smithi079.stdout:100: finishing write tid 1 to smithi07929762-27 2024-04-24T01:55:39.524 INFO:tasks.rados.rados.0.smithi079.stdout:105: expect (ObjNum 61 snap 2 seq_num 61) 2024-04-24T01:55:39.679 INFO:tasks.rados.rados.0.smithi079.stdout:101: expect (ObjNum 33 snap 0 seq_num 33) 2024-04-24T01:55:39.710 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T01:55:39.780 INFO:tasks.rados.rados.0.smithi079.stdout:100: finishing write tid 2 to smithi07929762-27 2024-04-24T01:55:39.780 INFO:tasks.rados.rados.0.smithi079.stdout:103: expect (ObjNum 18 snap 0 seq_num 18) 2024-04-24T01:55:39.815 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:39.813+0000 7f171be6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:39.819 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:39.818+0000 7f171be6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:39.880 INFO:tasks.rados.rados.0.smithi079.stdout:100: finishing write tid 3 to smithi07929762-27 2024-04-24T01:55:39.880 INFO:tasks.rados.rados.0.smithi079.stdout:100: finishing write tid 4 to smithi07929762-27 2024-04-24T01:55:39.880 INFO:tasks.rados.rados.0.smithi079.stdout:100: finishing write tid 5 to smithi07929762-27 2024-04-24T01:55:39.880 INFO:tasks.rados.rados.0.smithi079.stdout:100: oid 27 updating version 0 to 11 2024-04-24T01:55:39.881 INFO:tasks.rados.rados.0.smithi079.stdout:100: oid 27 version 11 is already newer than 11 2024-04-24T01:55:39.881 INFO:tasks.rados.rados.0.smithi079.stdout:100: oid 27 version 11 is already newer than 11 2024-04-24T01:55:39.881 INFO:tasks.rados.rados.0.smithi079.stdout:100: oid 27 version 11 is already newer than 11 2024-04-24T01:55:39.881 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 27 v 11 (ObjNum 74 snap 3 seq_num 74) dirty exists 2024-04-24T01:55:39.881 INFO:tasks.rados.rados.0.smithi079.stdout:100: left oid 27 (ObjNum 74 snap 3 seq_num 74) 2024-04-24T01:55:39.881 INFO:tasks.rados.rados.0.smithi079.stdout:104: expect (ObjNum 34 snap 0 seq_num 34) 2024-04-24T01:55:40.234 INFO:teuthology.orchestra.run.smithi079.stderr:reweighted osd.0 to 1 (10000) 2024-04-24T01:55:40.239 INFO:tasks.rados.rados.0.smithi079.stdout:100: done (6 left) 2024-04-24T01:55:40.239 INFO:tasks.rados.rados.0.smithi079.stdout:101: done (5 left) 2024-04-24T01:55:40.239 INFO:tasks.rados.rados.0.smithi079.stdout:102: done (4 left) 2024-04-24T01:55:40.239 INFO:tasks.rados.rados.0.smithi079.stdout:103: done (3 left) 2024-04-24T01:55:40.239 INFO:tasks.rados.rados.0.smithi079.stdout:104: done (2 left) 2024-04-24T01:55:40.239 INFO:tasks.rados.rados.0.smithi079.stdout:105: done (1 left) 2024-04-24T01:55:40.239 INFO:tasks.rados.rados.0.smithi079.stdout:108: done (0 left) 2024-04-24T01:55:40.239 INFO:tasks.rados.rados.0.smithi079.stdout:109: delete oid 42 current snap is 4 2024-04-24T01:55:40.249 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-04-24T01:55:40.297 INFO:tasks.rados.rados.0.smithi079.stdout:109: done (0 left) 2024-04-24T01:55:40.297 INFO:tasks.rados.rados.0.smithi079.stdout:110: read oid 48 snap 3 2024-04-24T01:55:40.297 INFO:tasks.rados.rados.0.smithi079.stdout:110: expect (ObjNum 47 snap 0 seq_num 47) 2024-04-24T01:55:40.297 INFO:tasks.rados.rados.0.smithi079.stdout:111: snap_create 2024-04-24T01:55:40.354 INFO:tasks.rados.rados.0.smithi079.stdout:110: expect (ObjNum 47 snap 0 seq_num 47) 2024-04-24T01:55:40.377 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:40.371+0000 7f59818ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:40.381 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:40.381+0000 7f59818ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:41.241 INFO:teuthology.orchestra.run.smithi079.stderr:reweighted osd.1 to 1 (10000) 2024-04-24T01:55:41.249 INFO:tasks.rados.rados.0.smithi079.stdout:110: done (1 left) 2024-04-24T01:55:41.249 INFO:tasks.rados.rados.0.smithi079.stdout:111: done (0 left) 2024-04-24T01:55:41.250 INFO:tasks.rados.rados.0.smithi079.stdout:112: snap_create 2024-04-24T01:55:41.256 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-04-24T01:55:41.367 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:41.360+0000 7fa9e655d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:41.370 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:41.370+0000 7fa9e655d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:41.926 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T01:55:42.033 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:42.031+0000 7f808346d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:42.037 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:42.036+0000 7f808346d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:42.241 INFO:teuthology.orchestra.run.smithi079.stderr:reweighted osd.2 to 1 (10000) 2024-04-24T01:55:42.247 INFO:tasks.rados.rados.0.smithi079.stdout:112: done (0 left) 2024-04-24T01:55:42.247 INFO:tasks.rados.rados.0.smithi079.stdout:113: read oid 35 snap -1 2024-04-24T01:55:42.247 INFO:tasks.rados.rados.0.smithi079.stdout:113: expect (ObjNum 34 snap 0 seq_num 34) 2024-04-24T01:55:42.247 INFO:tasks.rados.rados.0.smithi079.stdout:114: read oid 9 snap -1 2024-04-24T01:55:42.248 INFO:tasks.rados.rados.0.smithi079.stdout:114: expect (ObjNum 8 snap 0 seq_num 8) 2024-04-24T01:55:42.248 INFO:tasks.rados.rados.0.smithi079.stdout:115: delete oid 26 current snap is 6 2024-04-24T01:55:42.258 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-04-24T01:55:42.302 INFO:tasks.rados.rados.0.smithi079.stdout:113: expect (ObjNum 34 snap 0 seq_num 34) 2024-04-24T01:55:42.389 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:42.383+0000 7fd328cad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:42.393 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:42.392+0000 7fd328cad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:42.406 INFO:tasks.rados.rados.0.smithi079.stdout:114: expect (ObjNum 8 snap 0 seq_num 8) 2024-04-24T01:55:42.488 INFO:tasks.rados.rados.0.smithi079.stdout:113: done (2 left) 2024-04-24T01:55:42.488 INFO:tasks.rados.rados.0.smithi079.stdout:114: done (1 left) 2024-04-24T01:55:42.488 INFO:tasks.rados.rados.0.smithi079.stdout:115: done (0 left) 2024-04-24T01:55:42.488 INFO:tasks.rados.rados.0.smithi079.stdout:116: write oid 28 current snap is 6 2024-04-24T01:55:42.488 INFO:tasks.rados.rados.0.smithi079.stdout:116: seq_num 78 ranges {510946=632952,1798066=477679,2742758=457217} 2024-04-24T01:55:42.493 INFO:tasks.rados.rados.0.smithi079.stdout:116: writing smithi07929762-28 from 510946 to 1143898 tid 1 2024-04-24T01:55:42.500 INFO:tasks.rados.rados.0.smithi079.stdout:116: writing smithi07929762-28 from 1798066 to 2275745 tid 2 2024-04-24T01:55:42.507 INFO:tasks.rados.rados.0.smithi079.stdout:116: writing smithi07929762-28 from 2742758 to 3199975 tid 3 2024-04-24T01:55:42.507 INFO:tasks.rados.rados.0.smithi079.stdout:117: write oid 50 current snap is 6 2024-04-24T01:55:42.507 INFO:tasks.rados.rados.0.smithi079.stdout:117: seq_num 79 ranges {527753=708201,1856548=737690,3286534=41219} 2024-04-24T01:55:42.512 INFO:tasks.rados.rados.0.smithi079.stdout:117: writing smithi07929762-50 from 527753 to 1235954 tid 1 2024-04-24T01:55:42.519 INFO:tasks.rados.rados.0.smithi079.stdout:117: writing smithi07929762-50 from 1856548 to 2594238 tid 2 2024-04-24T01:55:42.524 INFO:tasks.rados.rados.0.smithi079.stdout:117: writing smithi07929762-50 from 3286534 to 3327753 tid 3 2024-04-24T01:55:42.524 INFO:tasks.rados.rados.0.smithi079.stdout:118: write oid 49 current snap is 6 2024-04-24T01:55:42.524 INFO:tasks.rados.rados.0.smithi079.stdout:118: seq_num 80 ranges {544560=783450,1915030=481348,2797604=479761} 2024-04-24T01:55:42.531 INFO:tasks.rados.rados.0.smithi079.stdout:118: writing smithi07929762-49 from 544560 to 1328010 tid 1 2024-04-24T01:55:42.537 INFO:tasks.rados.rados.0.smithi079.stdout:118: writing smithi07929762-49 from 1915030 to 2396378 tid 2 2024-04-24T01:55:42.543 INFO:tasks.rados.rados.0.smithi079.stdout:118: writing smithi07929762-49 from 2797604 to 3277365 tid 3 2024-04-24T01:55:42.543 INFO:tasks.rados.rados.0.smithi079.stdout:119: write (excl) oid 26 current snap is 6 2024-04-24T01:55:42.543 INFO:tasks.rados.rados.0.smithi079.stdout:116: finishing write tid 1 to smithi07929762-28 2024-04-24T01:55:42.543 INFO:tasks.rados.rados.0.smithi079.stdout:119: seq_num 81 ranges {561367=458699,1573512=741359,2941380=419987} 2024-04-24T01:55:42.548 INFO:tasks.rados.rados.0.smithi079.stdout:119: writing smithi07929762-26 from 561367 to 1020066 tid 1 2024-04-24T01:55:42.557 INFO:tasks.rados.rados.0.smithi079.stdout:119: writing smithi07929762-26 from 1573512 to 2314871 tid 2 2024-04-24T01:55:42.563 INFO:tasks.rados.rados.0.smithi079.stdout:119: writing smithi07929762-26 from 2941380 to 3361367 tid 3 2024-04-24T01:55:42.563 INFO:tasks.rados.rados.0.smithi079.stdout:120: write (excl) oid 44 current snap is 6 2024-04-24T01:55:42.564 INFO:tasks.rados.rados.0.smithi079.stdout:120: seq_num 82 ranges {578174=533948,1515641=485017,2736097=502305} 2024-04-24T01:55:42.571 INFO:tasks.rados.rados.0.smithi079.stdout:120: writing smithi07929762-44 from 578174 to 1112122 tid 1 2024-04-24T01:55:42.578 INFO:tasks.rados.rados.0.smithi079.stdout:120: writing smithi07929762-44 from 1515641 to 2000658 tid 2 2024-04-24T01:55:42.589 INFO:tasks.rados.rados.0.smithi079.stdout:120: writing smithi07929762-44 from 2736097 to 3238402 tid 3 2024-04-24T01:55:42.589 INFO:tasks.rados.rados.0.smithi079.stdout:121: read oid 22 snap -1 2024-04-24T01:55:42.589 INFO:tasks.rados.rados.0.smithi079.stdout:121: expect (ObjNum 68 snap 3 seq_num 68) 2024-04-24T01:55:42.589 INFO:tasks.rados.rados.0.smithi079.stdout:122: snap_remove snap 5 2024-04-24T01:55:43.254 INFO:teuthology.orchestra.run.smithi079.stderr:reweighted osd.3 to 1 (10000) 2024-04-24T01:55:43.258 INFO:tasks.rados.rados.0.smithi079.stdout:122: done (6 left) 2024-04-24T01:55:43.258 INFO:tasks.rados.rados.0.smithi079.stdout:123: read oid 37 snap -1 2024-04-24T01:55:43.258 INFO:tasks.rados.rados.0.smithi079.stdout:123: expect (ObjNum 36 snap 0 seq_num 36) 2024-04-24T01:55:43.258 INFO:tasks.rados.rados.0.smithi079.stdout:124: snap_remove snap 4 2024-04-24T01:55:43.271 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-04-24T01:55:43.271 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T01:55:43.420 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:43.416+0000 7ffb00554640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:43.424 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:43.423+0000 7ffb00554640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:43.612 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:55:43.612 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-24T01:55:43.624 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":35,"stamp":"2024-04-24T01:55:43.255710+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":112473181,"num_objects":55,"num_object_clones":9,"num_object_copies":165,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":348,"num_read_kb":99087,"num_write":428,"num_write_kb":79810,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":368,"ondisk_log_size":368,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":289560,"kb_used_data":182264,"kb_used_omap":0,"kb_used_meta":107264,"kb_avail":377197800,"statfs":{"total":386547056640,"available":386250547200,"internally_reserved":0,"allocated":186638336,"data_stored":192826530,"data_compressed":89889,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109838336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-1747895,"num_objects":-1,"num_object_clones":0,"num_object_copies":-3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":10856,"num_write":7,"num_write_kb":1583,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.028144"},"pg_stats":[{"pgid":"3.f","version":"19'6","reported_seq":41,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T01:55:38.308025+0000","last_change":"2024-04-24T01:55:38.329711+0000","last_active":"2024-04-24T01:55:38.308025+0000","last_peered":"2024-04-24T01:55:38.308025+0000","last_clean":"2024-04-24T01:55:38.308025+0000","last_became_active":"2024-04-24T01:55:30.386726+0000","last_became_peered":"2024-04-24T01:55:30.386726+0000","last_unstale":"2024-04-24T01:55:38.308025+0000","last_undegraded":"2024-04-24T01:55:38.308025+0000","last_fullsized":"2024-04-24T01:55:38.308025+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5310946,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":5413,"num_write":8,"num_write_kb":2595,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"24'29","reported_seq":67,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T01:55:38.311461+0000","last_change":"2024-04-24T01:55:38.375272+0000","last_active":"2024-04-24T01:55:38.311461+0000","last_peered":"2024-04-24T01:55:38.311461+0000","last_clean":"2024-04-24T01:55:38.311461+0000","last_became_active":"2024-04-24T01:55:30.321866+0000","last_became_peered":"2024-04-24T01:55:30.321866+0000","last_unstale":"2024-04-24T01:55:38.311461+0000","last_undegraded":"2024-04-24T01:55:38.311461+0000","last_fullsized":"2024-04-24T01:55:38.311461+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0,"stat_sum":{"num_bytes":11445435,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":23,"num_read_kb":9709,"num_write":28,"num_write_kb":8340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"26'27","reported_seq":72,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-24T01:55:42.262416+0000","last_change":"2024-04-24T01:55:39.343454+0000","last_active":"2024-04-24T01:55:42.262416+0000","last_peered":"2024-04-24T01:55:42.262416+0000","last_clean":"2024-04-24T01:55:42.262416+0000","last_became_active":"2024-04-24T01:55:30.320186+0000","last_became_peered":"2024-04-24T01:55:30.320186+0000","last_unstale":"2024-04-24T01:55:42.262416+0000","last_undegraded":"2024-04-24T01:55:42.262416+0000","last_fullsized":"2024-04-24T01:55:42.262416+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0,"stat_sum":{"num_bytes":5092455,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":14,"num_read_kb":6,"num_write":27,"num_write_kb":6415,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"24'24","reported_seq":57,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T01:55:38.355949+0000","last_change":"2024-04-24T01:55:38.422353+0000","last_active":"2024-04-24T01:55:38.355949+0000","last_peered":"2024-04-24T01:55:38.355949+0000","last_clean":"2024-04-24T01:55:38.355949+0000","last_became_active":"2024-04-24T01:55:30.288081+0000","last_became_peered":"2024-04-24T01:55:30.288081+0000","last_unstale":"2024-04-24T01:55:38.355949+0000","last_undegraded":"2024-04-24T01:55:38.355949+0000","last_fullsized":"2024-04-24T01:55:38.355949+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0,"stat_sum":{"num_bytes":8420208,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":16,"num_read_kb":5859,"num_write":21,"num_write_kb":6440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"19'7","reported_seq":45,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T01:55:38.257911+0000","last_change":"2024-04-24T01:55:38.262585+0000","last_active":"2024-04-24T01:55:38.257911+0000","last_peered":"2024-04-24T01:55:38.257911+0000","last_clean":"2024-04-24T01:55:38.257911+0000","last_became_active":"2024-04-24T01:55:30.333815+0000","last_became_peered":"2024-04-24T01:55:30.333815+0000","last_unstale":"2024-04-24T01:55:38.257911+0000","last_undegraded":"2024-04-24T01:55:38.257911+0000","last_fullsized":"2024-04-24T01:55:38.257911+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2588245,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":17,"num_read_kb":11069,"num_write":9,"num_write_kb":2420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"24'35","reported_seq":72,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T01:55:38.401002+0000","last_change":"2024-04-24T01:55:38.429985+0000","last_active":"2024-04-24T01:55:38.401002+0000","last_peered":"2024-04-24T01:55:38.401002+0000","last_clean":"2024-04-24T01:55:38.401002+0000","last_became_active":"2024-04-24T01:55:30.315075+0000","last_became_peered":"2024-04-24T01:55:30.315075+0000","last_unstale":"2024-04-24T01:55:38.401002+0000","last_undegraded":"2024-04-24T01:55:38.401002+0000","last_fullsized":"2024-04-24T01:55:38.401002+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":3,"snaptrim_duration":0,"stat_sum":{"num_bytes":14840383,"num_objects":7,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":24,"num_read_kb":5287,"num_write":37,"num_write_kb":10374,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"19'7","reported_seq":39,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T01:55:38.327281+0000","last_change":"2024-04-24T01:55:38.335187+0000","last_active":"2024-04-24T01:55:38.327281+0000","last_peered":"2024-04-24T01:55:38.327281+0000","last_clean":"2024-04-24T01:55:38.327281+0000","last_became_active":"2024-04-24T01:55:30.350300+0000","last_became_peered":"2024-04-24T01:55:30.350300+0000","last_unstale":"2024-04-24T01:55:38.327281+0000","last_undegraded":"2024-04-24T01:55:38.327281+0000","last_fullsized":"2024-04-24T01:55:38.327281+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2050421,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":9,"num_write_kb":2009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"18'4","reported_seq":48,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-24T01:55:42.276127+0000","last_change":"2024-04-24T01:55:39.302818+0000","last_active":"2024-04-24T01:55:42.276127+0000","last_peered":"2024-04-24T01:55:42.276127+0000","last_clean":"2024-04-24T01:55:42.276127+0000","last_became_active":"2024-04-24T01:55:30.376813+0000","last_became_peered":"2024-04-24T01:55:30.376813+0000","last_unstale":"2024-04-24T01:55:42.276127+0000","last_undegraded":"2024-04-24T01:55:42.276127+0000","last_fullsized":"2024-04-24T01:55:42.276127+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'97","reported_seq":170,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T01:55:38.251696+0000","last_change":"2024-04-24T01:55:16.196470+0000","last_active":"2024-04-24T01:55:38.251696+0000","last_peered":"2024-04-24T01:55:38.251696+0000","last_clean":"2024-04-24T01:55:38.251696+0000","last_became_active":"2024-04-24T01:55:16.190464+0000","last_became_peered":"2024-04-24T01:55:16.190464+0000","last_unstale":"2024-04-24T01:55:38.251696+0000","last_undegraded":"2024-04-24T01:55:38.251696+0000","last_fullsized":"2024-04-24T01:55:38.251696+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:15.127115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:15.127115+0000","last_clean_scrub_stamp":"2024-04-24T01:55:15.127115+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":49,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-24T01:55:42.283481+0000","last_change":"2024-04-24T01:55:18.236148+0000","last_active":"2024-04-24T01:55:42.283481+0000","last_peered":"2024-04-24T01:55:42.283481+0000","last_clean":"2024-04-24T01:55:42.283481+0000","last_became_active":"2024-04-24T01:55:18.230154+0000","last_became_peered":"2024-04-24T01:55:18.230154+0000","last_unstale":"2024-04-24T01:55:42.283481+0000","last_undegraded":"2024-04-24T01:55:42.283481+0000","last_fullsized":"2024-04-24T01:55:42.283481+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"28'22","reported_seq":82,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-24T01:55:42.621273+0000","last_change":"2024-04-24T01:55:39.300151+0000","last_active":"2024-04-24T01:55:42.621273+0000","last_peered":"2024-04-24T01:55:42.621273+0000","last_clean":"2024-04-24T01:55:42.621273+0000","last_became_active":"2024-04-24T01:55:30.316222+0000","last_became_peered":"2024-04-24T01:55:30.316222+0000","last_unstale":"2024-04-24T01:55:42.621273+0000","last_undegraded":"2024-04-24T01:55:42.621273+0000","last_fullsized":"2024-04-24T01:55:42.621273+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11243751,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":20531,"num_write":27,"num_write_kb":7302,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":37,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T01:55:38.322162+0000","last_change":"2024-04-24T01:55:18.265905+0000","last_active":"2024-04-24T01:55:38.322162+0000","last_peered":"2024-04-24T01:55:38.322162+0000","last_clean":"2024-04-24T01:55:38.322162+0000","last_became_active":"2024-04-24T01:55:18.257020+0000","last_became_peered":"2024-04-24T01:55:18.257020+0000","last_unstale":"2024-04-24T01:55:38.322162+0000","last_undegraded":"2024-04-24T01:55:38.322162+0000","last_fullsized":"2024-04-24T01:55:38.322162+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"19'13","reported_seq":47,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T01:55:38.244971+0000","last_change":"2024-04-24T01:55:38.259790+0000","last_active":"2024-04-24T01:55:38.244971+0000","last_peered":"2024-04-24T01:55:38.244971+0000","last_clean":"2024-04-24T01:55:38.244971+0000","last_became_active":"2024-04-24T01:55:30.344129+0000","last_became_peered":"2024-04-24T01:55:30.344129+0000","last_unstale":"2024-04-24T01:55:38.244971+0000","last_undegraded":"2024-04-24T01:55:38.244971+0000","last_fullsized":"2024-04-24T01:55:38.244971+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":17,"num_write_kb":4517,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":49,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-24T01:55:42.265793+0000","last_change":"2024-04-24T01:55:18.215251+0000","last_active":"2024-04-24T01:55:42.265793+0000","last_peered":"2024-04-24T01:55:42.265793+0000","last_clean":"2024-04-24T01:55:42.265793+0000","last_became_active":"2024-04-24T01:55:18.206605+0000","last_became_peered":"2024-04-24T01:55:18.206605+0000","last_unstale":"2024-04-24T01:55:42.265793+0000","last_undegraded":"2024-04-24T01:55:42.265793+0000","last_fullsized":"2024-04-24T01:55:42.265793+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"24'13","reported_seq":44,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T01:55:38.264160+0000","last_change":"2024-04-24T01:55:38.301424+0000","last_active":"2024-04-24T01:55:38.264160+0000","last_peered":"2024-04-24T01:55:38.264160+0000","last_clean":"2024-04-24T01:55:38.264160+0000","last_became_active":"2024-04-24T01:55:30.401366+0000","last_became_peered":"2024-04-24T01:55:30.401366+0000","last_unstale":"2024-04-24T01:55:38.264160+0000","last_undegraded":"2024-04-24T01:55:38.264160+0000","last_fullsized":"2024-04-24T01:55:38.264160+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5815156,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":13,"num_write_kb":3300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":37,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T01:55:38.321580+0000","last_change":"2024-04-24T01:55:18.230893+0000","last_active":"2024-04-24T01:55:38.321580+0000","last_peered":"2024-04-24T01:55:38.321580+0000","last_clean":"2024-04-24T01:55:38.321580+0000","last_became_active":"2024-04-24T01:55:18.225188+0000","last_became_peered":"2024-04-24T01:55:18.225188+0000","last_unstale":"2024-04-24T01:55:38.321580+0000","last_undegraded":"2024-04-24T01:55:38.321580+0000","last_fullsized":"2024-04-24T01:55:38.321580+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"24'24","reported_seq":57,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T01:55:38.312277+0000","last_change":"2024-04-24T01:55:38.396718+0000","last_active":"2024-04-24T01:55:38.312277+0000","last_peered":"2024-04-24T01:55:38.312277+0000","last_clean":"2024-04-24T01:55:38.312277+0000","last_became_active":"2024-04-24T01:55:30.343318+0000","last_became_peered":"2024-04-24T01:55:30.343318+0000","last_unstale":"2024-04-24T01:55:38.312277+0000","last_undegraded":"2024-04-24T01:55:38.312277+0000","last_fullsized":"2024-04-24T01:55:38.312277+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0,"stat_sum":{"num_bytes":8823576,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":14,"num_read_kb":3687,"num_write":21,"num_write_kb":6024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"24'20","reported_seq":75,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-24T01:55:42.328175+0000","last_change":"2024-04-24T01:55:39.302023+0000","last_active":"2024-04-24T01:55:42.328175+0000","last_peered":"2024-04-24T01:55:42.328175+0000","last_clean":"2024-04-24T01:55:42.328175+0000","last_became_active":"2024-04-24T01:55:30.345308+0000","last_became_peered":"2024-04-24T01:55:30.345308+0000","last_unstale":"2024-04-24T01:55:42.328175+0000","last_undegraded":"2024-04-24T01:55:42.328175+0000","last_fullsized":"2024-04-24T01:55:42.328175+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0,"stat_sum":{"num_bytes":7462209,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":28,"num_read_kb":15438,"num_write":22,"num_write_kb":5191,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":37,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T01:55:38.290232+0000","last_change":"2024-04-24T01:55:18.224238+0000","last_active":"2024-04-24T01:55:38.290232+0000","last_peered":"2024-04-24T01:55:38.290232+0000","last_clean":"2024-04-24T01:55:38.290232+0000","last_became_active":"2024-04-24T01:55:18.219131+0000","last_became_peered":"2024-04-24T01:55:18.219131+0000","last_unstale":"2024-04-24T01:55:38.290232+0000","last_undegraded":"2024-04-24T01:55:38.290232+0000","last_fullsized":"2024-04-24T01:55:38.290232+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"19'7","reported_seq":39,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T01:55:38.289651+0000","last_change":"2024-04-24T01:55:38.299107+0000","last_active":"2024-04-24T01:55:38.289651+0000","last_peered":"2024-04-24T01:55:38.289651+0000","last_clean":"2024-04-24T01:55:38.289651+0000","last_became_active":"2024-04-24T01:55:30.319958+0000","last_became_peered":"2024-04-24T01:55:30.319958+0000","last_unstale":"2024-04-24T01:55:38.289651+0000","last_undegraded":"2024-04-24T01:55:38.289651+0000","last_fullsized":"2024-04-24T01:55:38.289651+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2840350,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":9,"num_write_kb":2414,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":37,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T01:55:38.259623+0000","last_change":"2024-04-24T01:55:18.235763+0000","last_active":"2024-04-24T01:55:38.259623+0000","last_peered":"2024-04-24T01:55:38.259623+0000","last_clean":"2024-04-24T01:55:38.259623+0000","last_became_active":"2024-04-24T01:55:18.229175+0000","last_became_peered":"2024-04-24T01:55:18.229175+0000","last_unstale":"2024-04-24T01:55:38.259623+0000","last_undegraded":"2024-04-24T01:55:38.259623+0000","last_fullsized":"2024-04-24T01:55:38.259623+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"24'11","reported_seq":52,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T01:55:38.312846+0000","last_change":"2024-04-24T01:55:38.260559+0000","last_active":"2024-04-24T01:55:38.312846+0000","last_peered":"2024-04-24T01:55:38.312846+0000","last_clean":"2024-04-24T01:55:38.312846+0000","last_became_active":"2024-04-24T01:55:30.372690+0000","last_became_peered":"2024-04-24T01:55:30.372690+0000","last_unstale":"2024-04-24T01:55:38.312846+0000","last_undegraded":"2024-04-24T01:55:38.312846+0000","last_fullsized":"2024-04-24T01:55:38.312846+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5714314,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":24,"num_read_kb":16699,"num_write":13,"num_write_kb":3488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":37,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T01:55:38.292109+0000","last_change":"2024-04-24T01:55:18.278351+0000","last_active":"2024-04-24T01:55:38.292109+0000","last_peered":"2024-04-24T01:55:38.292109+0000","last_clean":"2024-04-24T01:55:38.292109+0000","last_became_active":"2024-04-24T01:55:18.273668+0000","last_became_peered":"2024-04-24T01:55:18.273668+0000","last_unstale":"2024-04-24T01:55:38.292109+0000","last_undegraded":"2024-04-24T01:55:38.292109+0000","last_fullsized":"2024-04-24T01:55:38.292109+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"24'22","reported_seq":59,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T01:55:38.302148+0000","last_change":"2024-04-24T01:55:38.353130+0000","last_active":"2024-04-24T01:55:38.302148+0000","last_peered":"2024-04-24T01:55:38.302148+0000","last_clean":"2024-04-24T01:55:38.302148+0000","last_became_active":"2024-04-24T01:55:30.363123+0000","last_became_peered":"2024-04-24T01:55:30.363123+0000","last_unstale":"2024-04-24T01:55:38.302148+0000","last_undegraded":"2024-04-24T01:55:38.302148+0000","last_fullsized":"2024-04-24T01:55:38.302148+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12420208,"num_objects":6,"num_object_clones":1,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":17,"num_read_kb":5286,"num_write":25,"num_write_kb":6239,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":49,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-24T01:55:42.281521+0000","last_change":"2024-04-24T01:55:18.241117+0000","last_active":"2024-04-24T01:55:42.281521+0000","last_peered":"2024-04-24T01:55:42.281521+0000","last_clean":"2024-04-24T01:55:42.281521+0000","last_became_active":"2024-04-24T01:55:18.234313+0000","last_became_peered":"2024-04-24T01:55:18.234313+0000","last_unstale":"2024-04-24T01:55:42.281521+0000","last_undegraded":"2024-04-24T01:55:42.281521+0000","last_fullsized":"2024-04-24T01:55:42.281521+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":111882813,"num_objects":53,"num_object_clones":9,"num_object_copies":159,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":245,"num_read_kb":98996,"num_write":291,"num_write_kb":77916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":190603264,"data_stored":196680048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":271,"ondisk_log_size":271,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":1}],"osd_stats":[{"osd":3,"up_from":10,"seq":42949672985,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":79664,"kb_used_data":52840,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94292176,"statfs":{"total":96636764160,"available":96555188224,"internally_reserved":0,"allocated":54108160,"data_stored":56162157,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949672985,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":69692,"kb_used_data":42868,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94302148,"statfs":{"total":96636764160,"available":96565399552,"internally_reserved":0,"allocated":43896832,"data_stored":46646156,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705690,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":70064,"kb_used_data":43240,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94301776,"statfs":{"total":96636764160,"available":96565018624,"internally_reserved":0,"allocated":44277760,"data_stored":46191330,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705693,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":70140,"kb_used_data":43316,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94301700,"statfs":{"total":96636764160,"available":96564940800,"internally_reserved":0,"allocated":44355584,"data_stored":43826887,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":44015616,"data_stored":45535870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49106944,"data_stored":49646785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":43634688,"data_stored":45990696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53846016,"data_stored":55506697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T01:55:43.626 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T01:55:43.734 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:43.729+0000 7f91606ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:43.737 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:43.736+0000 7f91606ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:43.923 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:55:43.924 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-24T01:55:43.939 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":35,"stamp":"2024-04-24T01:55:43.255710+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":112473181,"num_objects":55,"num_object_clones":9,"num_object_copies":165,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":348,"num_read_kb":99087,"num_write":428,"num_write_kb":79810,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":368,"ondisk_log_size":368,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":289560,"kb_used_data":182264,"kb_used_omap":0,"kb_used_meta":107264,"kb_avail":377197800,"statfs":{"total":386547056640,"available":386250547200,"internally_reserved":0,"allocated":186638336,"data_stored":192826530,"data_compressed":89889,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109838336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-1747895,"num_objects":-1,"num_object_clones":0,"num_object_copies":-3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":10856,"num_write":7,"num_write_kb":1583,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.028144"},"pg_stats":[{"pgid":"3.f","version":"19'6","reported_seq":41,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T01:55:38.308025+0000","last_change":"2024-04-24T01:55:38.329711+0000","last_active":"2024-04-24T01:55:38.308025+0000","last_peered":"2024-04-24T01:55:38.308025+0000","last_clean":"2024-04-24T01:55:38.308025+0000","last_became_active":"2024-04-24T01:55:30.386726+0000","last_became_peered":"2024-04-24T01:55:30.386726+0000","last_unstale":"2024-04-24T01:55:38.308025+0000","last_undegraded":"2024-04-24T01:55:38.308025+0000","last_fullsized":"2024-04-24T01:55:38.308025+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5310946,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":5413,"num_write":8,"num_write_kb":2595,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"24'29","reported_seq":67,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T01:55:38.311461+0000","last_change":"2024-04-24T01:55:38.375272+0000","last_active":"2024-04-24T01:55:38.311461+0000","last_peered":"2024-04-24T01:55:38.311461+0000","last_clean":"2024-04-24T01:55:38.311461+0000","last_became_active":"2024-04-24T01:55:30.321866+0000","last_became_peered":"2024-04-24T01:55:30.321866+0000","last_unstale":"2024-04-24T01:55:38.311461+0000","last_undegraded":"2024-04-24T01:55:38.311461+0000","last_fullsized":"2024-04-24T01:55:38.311461+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0,"stat_sum":{"num_bytes":11445435,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":23,"num_read_kb":9709,"num_write":28,"num_write_kb":8340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"26'27","reported_seq":72,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-24T01:55:42.262416+0000","last_change":"2024-04-24T01:55:39.343454+0000","last_active":"2024-04-24T01:55:42.262416+0000","last_peered":"2024-04-24T01:55:42.262416+0000","last_clean":"2024-04-24T01:55:42.262416+0000","last_became_active":"2024-04-24T01:55:30.320186+0000","last_became_peered":"2024-04-24T01:55:30.320186+0000","last_unstale":"2024-04-24T01:55:42.262416+0000","last_undegraded":"2024-04-24T01:55:42.262416+0000","last_fullsized":"2024-04-24T01:55:42.262416+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0,"stat_sum":{"num_bytes":5092455,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":14,"num_read_kb":6,"num_write":27,"num_write_kb":6415,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"24'24","reported_seq":57,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T01:55:38.355949+0000","last_change":"2024-04-24T01:55:38.422353+0000","last_active":"2024-04-24T01:55:38.355949+0000","last_peered":"2024-04-24T01:55:38.355949+0000","last_clean":"2024-04-24T01:55:38.355949+0000","last_became_active":"2024-04-24T01:55:30.288081+0000","last_became_peered":"2024-04-24T01:55:30.288081+0000","last_unstale":"2024-04-24T01:55:38.355949+0000","last_undegraded":"2024-04-24T01:55:38.355949+0000","last_fullsized":"2024-04-24T01:55:38.355949+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0,"stat_sum":{"num_bytes":8420208,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":16,"num_read_kb":5859,"num_write":21,"num_write_kb":6440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"19'7","reported_seq":45,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T01:55:38.257911+0000","last_change":"2024-04-24T01:55:38.262585+0000","last_active":"2024-04-24T01:55:38.257911+0000","last_peered":"2024-04-24T01:55:38.257911+0000","last_clean":"2024-04-24T01:55:38.257911+0000","last_became_active":"2024-04-24T01:55:30.333815+0000","last_became_peered":"2024-04-24T01:55:30.333815+0000","last_unstale":"2024-04-24T01:55:38.257911+0000","last_undegraded":"2024-04-24T01:55:38.257911+0000","last_fullsized":"2024-04-24T01:55:38.257911+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2588245,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":17,"num_read_kb":11069,"num_write":9,"num_write_kb":2420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"24'35","reported_seq":72,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T01:55:38.401002+0000","last_change":"2024-04-24T01:55:38.429985+0000","last_active":"2024-04-24T01:55:38.401002+0000","last_peered":"2024-04-24T01:55:38.401002+0000","last_clean":"2024-04-24T01:55:38.401002+0000","last_became_active":"2024-04-24T01:55:30.315075+0000","last_became_peered":"2024-04-24T01:55:30.315075+0000","last_unstale":"2024-04-24T01:55:38.401002+0000","last_undegraded":"2024-04-24T01:55:38.401002+0000","last_fullsized":"2024-04-24T01:55:38.401002+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":3,"snaptrim_duration":0,"stat_sum":{"num_bytes":14840383,"num_objects":7,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":24,"num_read_kb":5287,"num_write":37,"num_write_kb":10374,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"19'7","reported_seq":39,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T01:55:38.327281+0000","last_change":"2024-04-24T01:55:38.335187+0000","last_active":"2024-04-24T01:55:38.327281+0000","last_peered":"2024-04-24T01:55:38.327281+0000","last_clean":"2024-04-24T01:55:38.327281+0000","last_became_active":"2024-04-24T01:55:30.350300+0000","last_became_peered":"2024-04-24T01:55:30.350300+0000","last_unstale":"2024-04-24T01:55:38.327281+0000","last_undegraded":"2024-04-24T01:55:38.327281+0000","last_fullsized":"2024-04-24T01:55:38.327281+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2050421,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":9,"num_write_kb":2009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"18'4","reported_seq":48,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-24T01:55:42.276127+0000","last_change":"2024-04-24T01:55:39.302818+0000","last_active":"2024-04-24T01:55:42.276127+0000","last_peered":"2024-04-24T01:55:42.276127+0000","last_clean":"2024-04-24T01:55:42.276127+0000","last_became_active":"2024-04-24T01:55:30.376813+0000","last_became_peered":"2024-04-24T01:55:30.376813+0000","last_unstale":"2024-04-24T01:55:42.276127+0000","last_undegraded":"2024-04-24T01:55:42.276127+0000","last_fullsized":"2024-04-24T01:55:42.276127+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'97","reported_seq":170,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T01:55:38.251696+0000","last_change":"2024-04-24T01:55:16.196470+0000","last_active":"2024-04-24T01:55:38.251696+0000","last_peered":"2024-04-24T01:55:38.251696+0000","last_clean":"2024-04-24T01:55:38.251696+0000","last_became_active":"2024-04-24T01:55:16.190464+0000","last_became_peered":"2024-04-24T01:55:16.190464+0000","last_unstale":"2024-04-24T01:55:38.251696+0000","last_undegraded":"2024-04-24T01:55:38.251696+0000","last_fullsized":"2024-04-24T01:55:38.251696+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:15.127115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:15.127115+0000","last_clean_scrub_stamp":"2024-04-24T01:55:15.127115+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":49,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-24T01:55:42.283481+0000","last_change":"2024-04-24T01:55:18.236148+0000","last_active":"2024-04-24T01:55:42.283481+0000","last_peered":"2024-04-24T01:55:42.283481+0000","last_clean":"2024-04-24T01:55:42.283481+0000","last_became_active":"2024-04-24T01:55:18.230154+0000","last_became_peered":"2024-04-24T01:55:18.230154+0000","last_unstale":"2024-04-24T01:55:42.283481+0000","last_undegraded":"2024-04-24T01:55:42.283481+0000","last_fullsized":"2024-04-24T01:55:42.283481+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"28'22","reported_seq":82,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-24T01:55:42.621273+0000","last_change":"2024-04-24T01:55:39.300151+0000","last_active":"2024-04-24T01:55:42.621273+0000","last_peered":"2024-04-24T01:55:42.621273+0000","last_clean":"2024-04-24T01:55:42.621273+0000","last_became_active":"2024-04-24T01:55:30.316222+0000","last_became_peered":"2024-04-24T01:55:30.316222+0000","last_unstale":"2024-04-24T01:55:42.621273+0000","last_undegraded":"2024-04-24T01:55:42.621273+0000","last_fullsized":"2024-04-24T01:55:42.621273+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11243751,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":20531,"num_write":27,"num_write_kb":7302,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":37,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T01:55:38.322162+0000","last_change":"2024-04-24T01:55:18.265905+0000","last_active":"2024-04-24T01:55:38.322162+0000","last_peered":"2024-04-24T01:55:38.322162+0000","last_clean":"2024-04-24T01:55:38.322162+0000","last_became_active":"2024-04-24T01:55:18.257020+0000","last_became_peered":"2024-04-24T01:55:18.257020+0000","last_unstale":"2024-04-24T01:55:38.322162+0000","last_undegraded":"2024-04-24T01:55:38.322162+0000","last_fullsized":"2024-04-24T01:55:38.322162+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"19'13","reported_seq":47,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T01:55:38.244971+0000","last_change":"2024-04-24T01:55:38.259790+0000","last_active":"2024-04-24T01:55:38.244971+0000","last_peered":"2024-04-24T01:55:38.244971+0000","last_clean":"2024-04-24T01:55:38.244971+0000","last_became_active":"2024-04-24T01:55:30.344129+0000","last_became_peered":"2024-04-24T01:55:30.344129+0000","last_unstale":"2024-04-24T01:55:38.244971+0000","last_undegraded":"2024-04-24T01:55:38.244971+0000","last_fullsized":"2024-04-24T01:55:38.244971+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":17,"num_write_kb":4517,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":49,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-24T01:55:42.265793+0000","last_change":"2024-04-24T01:55:18.215251+0000","last_active":"2024-04-24T01:55:42.265793+0000","last_peered":"2024-04-24T01:55:42.265793+0000","last_clean":"2024-04-24T01:55:42.265793+0000","last_became_active":"2024-04-24T01:55:18.206605+0000","last_became_peered":"2024-04-24T01:55:18.206605+0000","last_unstale":"2024-04-24T01:55:42.265793+0000","last_undegraded":"2024-04-24T01:55:42.265793+0000","last_fullsized":"2024-04-24T01:55:42.265793+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"24'13","reported_seq":44,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T01:55:38.264160+0000","last_change":"2024-04-24T01:55:38.301424+0000","last_active":"2024-04-24T01:55:38.264160+0000","last_peered":"2024-04-24T01:55:38.264160+0000","last_clean":"2024-04-24T01:55:38.264160+0000","last_became_active":"2024-04-24T01:55:30.401366+0000","last_became_peered":"2024-04-24T01:55:30.401366+0000","last_unstale":"2024-04-24T01:55:38.264160+0000","last_undegraded":"2024-04-24T01:55:38.264160+0000","last_fullsized":"2024-04-24T01:55:38.264160+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5815156,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":13,"num_write_kb":3300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":37,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T01:55:38.321580+0000","last_change":"2024-04-24T01:55:18.230893+0000","last_active":"2024-04-24T01:55:38.321580+0000","last_peered":"2024-04-24T01:55:38.321580+0000","last_clean":"2024-04-24T01:55:38.321580+0000","last_became_active":"2024-04-24T01:55:18.225188+0000","last_became_peered":"2024-04-24T01:55:18.225188+0000","last_unstale":"2024-04-24T01:55:38.321580+0000","last_undegraded":"2024-04-24T01:55:38.321580+0000","last_fullsized":"2024-04-24T01:55:38.321580+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"24'24","reported_seq":57,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T01:55:38.312277+0000","last_change":"2024-04-24T01:55:38.396718+0000","last_active":"2024-04-24T01:55:38.312277+0000","last_peered":"2024-04-24T01:55:38.312277+0000","last_clean":"2024-04-24T01:55:38.312277+0000","last_became_active":"2024-04-24T01:55:30.343318+0000","last_became_peered":"2024-04-24T01:55:30.343318+0000","last_unstale":"2024-04-24T01:55:38.312277+0000","last_undegraded":"2024-04-24T01:55:38.312277+0000","last_fullsized":"2024-04-24T01:55:38.312277+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0,"stat_sum":{"num_bytes":8823576,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":14,"num_read_kb":3687,"num_write":21,"num_write_kb":6024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"24'20","reported_seq":75,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-24T01:55:42.328175+0000","last_change":"2024-04-24T01:55:39.302023+0000","last_active":"2024-04-24T01:55:42.328175+0000","last_peered":"2024-04-24T01:55:42.328175+0000","last_clean":"2024-04-24T01:55:42.328175+0000","last_became_active":"2024-04-24T01:55:30.345308+0000","last_became_peered":"2024-04-24T01:55:30.345308+0000","last_unstale":"2024-04-24T01:55:42.328175+0000","last_undegraded":"2024-04-24T01:55:42.328175+0000","last_fullsized":"2024-04-24T01:55:42.328175+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0,"stat_sum":{"num_bytes":7462209,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":28,"num_read_kb":15438,"num_write":22,"num_write_kb":5191,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":37,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T01:55:38.290232+0000","last_change":"2024-04-24T01:55:18.224238+0000","last_active":"2024-04-24T01:55:38.290232+0000","last_peered":"2024-04-24T01:55:38.290232+0000","last_clean":"2024-04-24T01:55:38.290232+0000","last_became_active":"2024-04-24T01:55:18.219131+0000","last_became_peered":"2024-04-24T01:55:18.219131+0000","last_unstale":"2024-04-24T01:55:38.290232+0000","last_undegraded":"2024-04-24T01:55:38.290232+0000","last_fullsized":"2024-04-24T01:55:38.290232+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"19'7","reported_seq":39,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T01:55:38.289651+0000","last_change":"2024-04-24T01:55:38.299107+0000","last_active":"2024-04-24T01:55:38.289651+0000","last_peered":"2024-04-24T01:55:38.289651+0000","last_clean":"2024-04-24T01:55:38.289651+0000","last_became_active":"2024-04-24T01:55:30.319958+0000","last_became_peered":"2024-04-24T01:55:30.319958+0000","last_unstale":"2024-04-24T01:55:38.289651+0000","last_undegraded":"2024-04-24T01:55:38.289651+0000","last_fullsized":"2024-04-24T01:55:38.289651+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2840350,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":9,"num_write_kb":2414,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":37,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T01:55:38.259623+0000","last_change":"2024-04-24T01:55:18.235763+0000","last_active":"2024-04-24T01:55:38.259623+0000","last_peered":"2024-04-24T01:55:38.259623+0000","last_clean":"2024-04-24T01:55:38.259623+0000","last_became_active":"2024-04-24T01:55:18.229175+0000","last_became_peered":"2024-04-24T01:55:18.229175+0000","last_unstale":"2024-04-24T01:55:38.259623+0000","last_undegraded":"2024-04-24T01:55:38.259623+0000","last_fullsized":"2024-04-24T01:55:38.259623+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"24'11","reported_seq":52,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T01:55:38.312846+0000","last_change":"2024-04-24T01:55:38.260559+0000","last_active":"2024-04-24T01:55:38.312846+0000","last_peered":"2024-04-24T01:55:38.312846+0000","last_clean":"2024-04-24T01:55:38.312846+0000","last_became_active":"2024-04-24T01:55:30.372690+0000","last_became_peered":"2024-04-24T01:55:30.372690+0000","last_unstale":"2024-04-24T01:55:38.312846+0000","last_undegraded":"2024-04-24T01:55:38.312846+0000","last_fullsized":"2024-04-24T01:55:38.312846+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5714314,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":24,"num_read_kb":16699,"num_write":13,"num_write_kb":3488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":37,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T01:55:38.292109+0000","last_change":"2024-04-24T01:55:18.278351+0000","last_active":"2024-04-24T01:55:38.292109+0000","last_peered":"2024-04-24T01:55:38.292109+0000","last_clean":"2024-04-24T01:55:38.292109+0000","last_became_active":"2024-04-24T01:55:18.273668+0000","last_became_peered":"2024-04-24T01:55:18.273668+0000","last_unstale":"2024-04-24T01:55:38.292109+0000","last_undegraded":"2024-04-24T01:55:38.292109+0000","last_fullsized":"2024-04-24T01:55:38.292109+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"24'22","reported_seq":59,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T01:55:38.302148+0000","last_change":"2024-04-24T01:55:38.353130+0000","last_active":"2024-04-24T01:55:38.302148+0000","last_peered":"2024-04-24T01:55:38.302148+0000","last_clean":"2024-04-24T01:55:38.302148+0000","last_became_active":"2024-04-24T01:55:30.363123+0000","last_became_peered":"2024-04-24T01:55:30.363123+0000","last_unstale":"2024-04-24T01:55:38.302148+0000","last_undegraded":"2024-04-24T01:55:38.302148+0000","last_fullsized":"2024-04-24T01:55:38.302148+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12420208,"num_objects":6,"num_object_clones":1,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":17,"num_read_kb":5286,"num_write":25,"num_write_kb":6239,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":49,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-24T01:55:42.281521+0000","last_change":"2024-04-24T01:55:18.241117+0000","last_active":"2024-04-24T01:55:42.281521+0000","last_peered":"2024-04-24T01:55:42.281521+0000","last_clean":"2024-04-24T01:55:42.281521+0000","last_became_active":"2024-04-24T01:55:18.234313+0000","last_became_peered":"2024-04-24T01:55:18.234313+0000","last_unstale":"2024-04-24T01:55:42.281521+0000","last_undegraded":"2024-04-24T01:55:42.281521+0000","last_fullsized":"2024-04-24T01:55:42.281521+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":111882813,"num_objects":53,"num_object_clones":9,"num_object_copies":159,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":245,"num_read_kb":98996,"num_write":291,"num_write_kb":77916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":190603264,"data_stored":196680048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":271,"ondisk_log_size":271,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":1}],"osd_stats":[{"osd":3,"up_from":10,"seq":42949672985,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":79664,"kb_used_data":52840,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94292176,"statfs":{"total":96636764160,"available":96555188224,"internally_reserved":0,"allocated":54108160,"data_stored":56162157,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949672985,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":69692,"kb_used_data":42868,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94302148,"statfs":{"total":96636764160,"available":96565399552,"internally_reserved":0,"allocated":43896832,"data_stored":46646156,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705690,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":70064,"kb_used_data":43240,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94301776,"statfs":{"total":96636764160,"available":96565018624,"internally_reserved":0,"allocated":44277760,"data_stored":46191330,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705693,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":70140,"kb_used_data":43316,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94301700,"statfs":{"total":96636764160,"available":96564940800,"internally_reserved":0,"allocated":44355584,"data_stored":43826887,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":44015616,"data_stored":45535870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49106944,"data_stored":49646785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":43634688,"data_stored":45990696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53846016,"data_stored":55506697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T01:55:43.941 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-04-24T01:55:43.941 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-04-24T01:55:43.941 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-04-24T01:55:43.941 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-04-24T01:55:44.049 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:44.043+0000 7f7f0e39d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:44.052 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:44.052+0000 7f7f0e39d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:44.068 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T01:55:44.068 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T01:55:44.068 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T01:55:44.068 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T01:55:44.068 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T01:55:44.068 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T01:55:44.068 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T01:55:44.068 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T01:55:44.068 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T01:55:44.068 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T01:55:44.068 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T01:55:44.069 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T01:55:44.071 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T01:55:44.071 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T01:55:44.072 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-04-24T01:55:44.138 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T01:55:44.180 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:44.174+0000 7f19ecead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:44.184 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:44.183+0000 7f19ecead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:44.201 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T01:55:44.201 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T01:55:44.201 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T01:55:44.201 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T01:55:44.201 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T01:55:44.201 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T01:55:44.201 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T01:55:44.201 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T01:55:44.202 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T01:55:44.202 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T01:55:44.202 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T01:55:44.202 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T01:55:44.204 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T01:55:44.205 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T01:55:44.205 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-04-24T01:55:44.253 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:44.251+0000 7fbd8a42d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:44.260 INFO:tasks.rados.rados.0.smithi079.stdout:124: done (7 left) 2024-04-24T01:55:44.260 INFO:tasks.rados.rados.0.smithi079.stdout:125: snap_remove snap 3 2024-04-24T01:55:44.262 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:44.261+0000 7fbd891cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:44.370 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:44.365+0000 7f00fa354640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:44.374 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:44.373+0000 7f00fa354640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:44.390 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T01:55:44.390 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T01:55:44.390 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T01:55:44.390 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T01:55:44.390 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T01:55:44.390 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T01:55:44.390 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T01:55:44.391 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T01:55:44.391 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T01:55:44.391 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T01:55:44.391 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T01:55:44.391 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T01:55:44.393 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T01:55:44.394 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T01:55:44.394 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-04-24T01:55:44.539 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:44.534+0000 7f8cbe42d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:44.543 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:44.542+0000 7f8cbe42d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:44.559 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T01:55:44.559 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T01:55:44.560 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T01:55:44.560 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T01:55:44.560 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T01:55:44.560 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T01:55:44.560 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T01:55:44.560 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T01:55:44.560 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T01:55:44.560 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T01:55:44.560 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T01:55:44.560 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T01:55:44.563 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T01:55:44.563 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T01:55:45.258 INFO:tasks.rados.rados.0.smithi079.stdout:125: done (7 left) 2024-04-24T01:55:45.258 INFO:tasks.rados.rados.0.smithi079.stdout:126: write oid 36 current snap is 6 2024-04-24T01:55:45.258 INFO:tasks.rados.rados.0.smithi079.stdout:116: finishing write tid 2 to smithi07929762-28 2024-04-24T01:55:45.258 INFO:tasks.rados.rados.0.smithi079.stdout:126: seq_num 83 ranges {594981=609197,1974123=745028,3279873=115108} 2024-04-24T01:55:45.264 INFO:tasks.rados.rados.0.smithi079.stdout:126: writing smithi07929762-36 from 594981 to 1204178 tid 1 2024-04-24T01:55:45.272 INFO:tasks.rados.rados.0.smithi079.stdout:126: writing smithi07929762-36 from 1974123 to 2719151 tid 2 2024-04-24T01:55:45.276 INFO:tasks.rados.rados.0.smithi079.stdout:126: writing smithi07929762-36 from 3279873 to 3394981 tid 3 2024-04-24T01:55:45.276 INFO:tasks.rados.rados.0.smithi079.stdout:127: write (excl) oid 35 current snap is 6 2024-04-24T01:55:45.276 INFO:tasks.rados.rados.0.smithi079.stdout:127: seq_num 84 ranges {611788=400799,1748958=488686,2907296=504492} 2024-04-24T01:55:45.281 INFO:tasks.rados.rados.0.smithi079.stdout:127: writing smithi07929762-35 from 611788 to 1012587 tid 1 2024-04-24T01:55:45.288 INFO:tasks.rados.rados.0.smithi079.stdout:127: writing smithi07929762-35 from 1748958 to 2237644 tid 2 2024-04-24T01:55:45.295 INFO:tasks.rados.rados.0.smithi079.stdout:127: writing smithi07929762-35 from 2907296 to 3411788 tid 3 2024-04-24T01:55:45.295 INFO:tasks.rados.rados.0.smithi079.stdout:128: write oid 2 current snap is 6 2024-04-24T01:55:45.295 INFO:tasks.rados.rados.0.smithi079.stdout:116: finishing write tid 3 to smithi07929762-28 2024-04-24T01:55:45.295 INFO:tasks.rados.rados.0.smithi079.stdout:128: seq_num 85 ranges {628595=476048,1807440=632344,2934719=452474} 2024-04-24T01:55:45.300 INFO:tasks.rados.rados.0.smithi079.stdout:128: writing smithi07929762-2 from 628595 to 1104643 tid 1 2024-04-24T01:55:45.308 INFO:tasks.rados.rados.0.smithi079.stdout:128: writing smithi07929762-2 from 1807440 to 2439784 tid 2 2024-04-24T01:55:45.314 INFO:tasks.rados.rados.0.smithi079.stdout:128: writing smithi07929762-2 from 2934719 to 3387193 tid 3 2024-04-24T01:55:45.314 INFO:tasks.rados.rados.0.smithi079.stdout:129: read oid 9 snap -1 2024-04-24T01:55:45.314 INFO:tasks.rados.rados.0.smithi079.stdout:129: expect (ObjNum 8 snap 0 seq_num 8) 2024-04-24T01:55:45.314 INFO:tasks.rados.rados.0.smithi079.stdout:130: read oid 46 snap -1 2024-04-24T01:55:45.314 INFO:tasks.rados.rados.0.smithi079.stdout:130: expect (ObjNum 62 snap 2 seq_num 62) 2024-04-24T01:55:45.314 INFO:tasks.rados.rados.0.smithi079.stdout:131: write oid 1 current snap is 6 2024-04-24T01:55:45.314 INFO:tasks.rados.rados.0.smithi079.stdout:131: seq_num 86 ranges {645402=551297,1749569=492355,2845789=599613} 2024-04-24T01:55:45.320 INFO:tasks.rados.rados.0.smithi079.stdout:131: writing smithi07929762-1 from 645402 to 1196699 tid 1 2024-04-24T01:55:45.327 INFO:tasks.rados.rados.0.smithi079.stdout:131: writing smithi07929762-1 from 1749569 to 2241924 tid 2 2024-04-24T01:55:45.335 INFO:tasks.rados.rados.0.smithi079.stdout:131: writing smithi07929762-1 from 2845789 to 3445402 tid 3 2024-04-24T01:55:45.335 INFO:tasks.rados.rados.0.smithi079.stdout:132: read oid 19 snap -1 2024-04-24T01:55:45.335 INFO:tasks.rados.rados.0.smithi079.stdout:132: expect (ObjNum 18 snap 0 seq_num 18) 2024-04-24T01:55:45.335 INFO:tasks.rados.rados.0.smithi079.stdout:133: write oid 43 current snap is 6 2024-04-24T01:55:45.335 INFO:tasks.rados.rados.0.smithi079.stdout:116: finishing write tid 4 to smithi07929762-28 2024-04-24T01:55:45.335 INFO:tasks.rados.rados.0.smithi079.stdout:133: seq_num 87 ranges {662209=626546,1808051=636013,2873212=588997} 2024-04-24T01:55:45.342 INFO:tasks.rados.rados.0.smithi079.stdout:133: writing smithi07929762-43 from 662209 to 1288755 tid 1 2024-04-24T01:55:45.350 INFO:tasks.rados.rados.0.smithi079.stdout:133: writing smithi07929762-43 from 1808051 to 2444064 tid 2 2024-04-24T01:55:45.358 INFO:tasks.rados.rados.0.smithi079.stdout:133: writing smithi07929762-43 from 2873212 to 3462209 tid 3 2024-04-24T01:55:45.358 INFO:tasks.rados.rados.0.smithi079.stdout:134: snap_create 2024-04-24T01:55:45.358 INFO:tasks.rados.rados.0.smithi079.stdout:116: finishing write tid 5 to smithi07929762-28 2024-04-24T01:55:45.358 INFO:tasks.rados.rados.0.smithi079.stdout:116: oid 28 updating version 0 to 36 2024-04-24T01:55:45.358 INFO:tasks.rados.rados.0.smithi079.stdout:116: oid 28 version 36 is already newer than 36 2024-04-24T01:55:45.358 INFO:tasks.rados.rados.0.smithi079.stdout:116: oid 28 version 36 is already newer than 36 2024-04-24T01:55:45.358 INFO:tasks.rados.rados.0.smithi079.stdout:116: oid 28 version 36 is already newer than 36 2024-04-24T01:55:45.358 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 28 v 36 (ObjNum 78 snap 6 seq_num 78) dirty exists 2024-04-24T01:55:45.358 INFO:tasks.rados.rados.0.smithi079.stdout:116: left oid 28 (ObjNum 78 snap 6 seq_num 78) 2024-04-24T01:55:45.358 INFO:tasks.rados.rados.0.smithi079.stdout:118: finishing write tid 1 to smithi07929762-49 2024-04-24T01:55:45.358 INFO:tasks.rados.rados.0.smithi079.stdout:117: finishing write tid 1 to smithi07929762-50 2024-04-24T01:55:45.358 INFO:tasks.rados.rados.0.smithi079.stdout:119: finishing write tid 1 to smithi07929762-26 2024-04-24T01:55:45.359 INFO:tasks.rados.rados.0.smithi079.stdout:118: finishing write tid 2 to smithi07929762-49 2024-04-24T01:55:45.359 INFO:tasks.rados.rados.0.smithi079.stdout:118: finishing write tid 3 to smithi07929762-49 2024-04-24T01:55:45.359 INFO:tasks.rados.rados.0.smithi079.stdout:117: finishing write tid 2 to smithi07929762-50 2024-04-24T01:55:45.359 INFO:tasks.rados.rados.0.smithi079.stdout:118: finishing write tid 4 to smithi07929762-49 2024-04-24T01:55:45.359 INFO:tasks.rados.rados.0.smithi079.stdout:119: finishing write tid 2 to smithi07929762-26 2024-04-24T01:55:45.359 INFO:tasks.rados.rados.0.smithi079.stdout:118: finishing write tid 5 to smithi07929762-49 2024-04-24T01:55:45.359 INFO:tasks.rados.rados.0.smithi079.stdout:118: oid 49 updating version 0 to 41 2024-04-24T01:55:45.359 INFO:tasks.rados.rados.0.smithi079.stdout:118: oid 49 version 41 is already newer than 41 2024-04-24T01:55:45.359 INFO:tasks.rados.rados.0.smithi079.stdout:118: oid 49 version 41 is already newer than 41 2024-04-24T01:55:45.359 INFO:tasks.rados.rados.0.smithi079.stdout:118: oid 49 version 41 is already newer than 41 2024-04-24T01:55:45.359 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 49 v 41 (ObjNum 80 snap 6 seq_num 80) dirty exists 2024-04-24T01:55:45.359 INFO:tasks.rados.rados.0.smithi079.stdout:118: left oid 49 (ObjNum 80 snap 6 seq_num 80) 2024-04-24T01:55:45.359 INFO:tasks.rados.rados.0.smithi079.stdout:120: finishing write tid 1 to smithi07929762-44 2024-04-24T01:55:45.359 INFO:tasks.rados.rados.0.smithi079.stdout:117: finishing write tid 3 to smithi07929762-50 2024-04-24T01:55:45.359 INFO:tasks.rados.rados.0.smithi079.stdout:121: expect (ObjNum 68 snap 3 seq_num 68) 2024-04-24T01:55:45.561 INFO:tasks.rados.rados.0.smithi079.stdout:117: finishing write tid 4 to smithi07929762-50 2024-04-24T01:55:45.561 INFO:tasks.rados.rados.0.smithi079.stdout:117: finishing write tid 5 to smithi07929762-50 2024-04-24T01:55:45.561 INFO:tasks.rados.rados.0.smithi079.stdout:117: oid 50 updating version 0 to 31 2024-04-24T01:55:45.561 INFO:tasks.rados.rados.0.smithi079.stdout:117: oid 50 version 31 is already newer than 31 2024-04-24T01:55:45.561 INFO:tasks.rados.rados.0.smithi079.stdout:117: oid 50 version 31 is already newer than 31 2024-04-24T01:55:45.561 INFO:tasks.rados.rados.0.smithi079.stdout:117: oid 50 version 31 is already newer than 31 2024-04-24T01:55:45.561 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 50 v 31 (ObjNum 79 snap 6 seq_num 79) dirty exists 2024-04-24T01:55:45.561 INFO:tasks.rados.rados.0.smithi079.stdout:117: left oid 50 (ObjNum 79 snap 6 seq_num 79) 2024-04-24T01:55:45.561 INFO:tasks.rados.rados.0.smithi079.stdout:119: finishing write tid 3 to smithi07929762-26 2024-04-24T01:55:45.561 INFO:tasks.rados.rados.0.smithi079.stdout:119: finishing write tid 4 to smithi07929762-26 2024-04-24T01:55:45.561 INFO:tasks.rados.rados.0.smithi079.stdout:119: finishing write tid 5 to smithi07929762-26 2024-04-24T01:55:45.562 INFO:tasks.rados.rados.0.smithi079.stdout:119: oid 26 updating version 0 to 22 2024-04-24T01:55:45.562 INFO:tasks.rados.rados.0.smithi079.stdout:119: oid 26 version 22 is already newer than 22 2024-04-24T01:55:45.562 INFO:tasks.rados.rados.0.smithi079.stdout:119: oid 26 version 22 is already newer than 19 2024-04-24T01:55:45.562 INFO:tasks.rados.rados.0.smithi079.stdout:119: oid 26 version 22 is already newer than 22 2024-04-24T01:55:45.562 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 26 v 22 (ObjNum 81 snap 6 seq_num 81) dirty exists 2024-04-24T01:55:45.562 INFO:tasks.rados.rados.0.smithi079.stdout:119: left oid 26 (ObjNum 81 snap 6 seq_num 81) 2024-04-24T01:55:45.562 INFO:tasks.rados.rados.0.smithi079.stdout:120: finishing write tid 2 to smithi07929762-44 2024-04-24T01:55:45.562 INFO:tasks.rados.rados.0.smithi079.stdout:120: finishing write tid 3 to smithi07929762-44 2024-04-24T01:55:45.562 INFO:tasks.rados.rados.0.smithi079.stdout:120: finishing write tid 4 to smithi07929762-44 2024-04-24T01:55:45.562 INFO:tasks.rados.rados.0.smithi079.stdout:120: finishing write tid 5 to smithi07929762-44 2024-04-24T01:55:45.562 INFO:tasks.rados.rados.0.smithi079.stdout:120: oid 44 updating version 0 to 31 2024-04-24T01:55:45.562 INFO:tasks.rados.rados.0.smithi079.stdout:120: oid 44 version 31 is already newer than 28 2024-04-24T01:55:45.562 INFO:tasks.rados.rados.0.smithi079.stdout:120: oid 44 version 31 is already newer than 31 2024-04-24T01:55:45.562 INFO:tasks.rados.rados.0.smithi079.stdout:120: oid 44 version 31 is already newer than 31 2024-04-24T01:55:45.563 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 44 v 31 (ObjNum 82 snap 6 seq_num 82) dirty exists 2024-04-24T01:55:45.563 INFO:tasks.rados.rados.0.smithi079.stdout:120: left oid 44 (ObjNum 82 snap 6 seq_num 82) 2024-04-24T01:55:45.563 INFO:tasks.rados.rados.0.smithi079.stdout:123: expect (ObjNum 36 snap 0 seq_num 36) 2024-04-24T01:55:45.657 INFO:tasks.rados.rados.0.smithi079.stdout:126: finishing write tid 1 to smithi07929762-36 2024-04-24T01:55:45.657 INFO:tasks.rados.rados.0.smithi079.stdout:129: expect (ObjNum 8 snap 0 seq_num 8) 2024-04-24T01:55:45.736 INFO:tasks.rados.rados.0.smithi079.stdout:127: finishing write tid 1 to smithi07929762-35 2024-04-24T01:55:45.737 INFO:tasks.rados.rados.0.smithi079.stdout:131: finishing write tid 1 to smithi07929762-1 2024-04-24T01:55:45.737 INFO:tasks.rados.rados.0.smithi079.stdout:126: finishing write tid 2 to smithi07929762-36 2024-04-24T01:55:45.737 INFO:tasks.rados.rados.0.smithi079.stdout:131: finishing write tid 2 to smithi07929762-1 2024-04-24T01:55:45.737 INFO:tasks.rados.rados.0.smithi079.stdout:133: finishing write tid 1 to smithi07929762-43 2024-04-24T01:55:45.737 INFO:tasks.rados.rados.0.smithi079.stdout:128: finishing write tid 1 to smithi07929762-2 2024-04-24T01:55:45.737 INFO:tasks.rados.rados.0.smithi079.stdout:126: finishing write tid 3 to smithi07929762-36 2024-04-24T01:55:45.737 INFO:tasks.rados.rados.0.smithi079.stdout:133: finishing write tid 2 to smithi07929762-43 2024-04-24T01:55:45.737 INFO:tasks.rados.rados.0.smithi079.stdout:131: finishing write tid 3 to smithi07929762-1 2024-04-24T01:55:45.737 INFO:tasks.rados.rados.0.smithi079.stdout:126: finishing write tid 4 to smithi07929762-36 2024-04-24T01:55:45.737 INFO:tasks.rados.rados.0.smithi079.stdout:126: finishing write tid 5 to smithi07929762-36 2024-04-24T01:55:45.737 INFO:tasks.rados.rados.0.smithi079.stdout:126: oid 36 updating version 0 to 15 2024-04-24T01:55:45.737 INFO:tasks.rados.rados.0.smithi079.stdout:126: oid 36 version 15 is already newer than 15 2024-04-24T01:55:45.737 INFO:tasks.rados.rados.0.smithi079.stdout:126: oid 36 version 15 is already newer than 15 2024-04-24T01:55:45.737 INFO:tasks.rados.rados.0.smithi079.stdout:126: oid 36 version 15 is already newer than 15 2024-04-24T01:55:45.738 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 36 v 15 (ObjNum 83 snap 6 seq_num 83) dirty exists 2024-04-24T01:55:45.738 INFO:tasks.rados.rados.0.smithi079.stdout:126: left oid 36 (ObjNum 83 snap 6 seq_num 83) 2024-04-24T01:55:45.738 INFO:tasks.rados.rados.0.smithi079.stdout:127: finishing write tid 2 to smithi07929762-35 2024-04-24T01:55:45.738 INFO:tasks.rados.rados.0.smithi079.stdout:130: expect (ObjNum 62 snap 2 seq_num 62) 2024-04-24T01:55:45.891 INFO:tasks.rados.rados.0.smithi079.stdout:131: finishing write tid 4 to smithi07929762-1 2024-04-24T01:55:45.891 INFO:tasks.rados.rados.0.smithi079.stdout:133: finishing write tid 3 to smithi07929762-43 2024-04-24T01:55:45.891 INFO:tasks.rados.rados.0.smithi079.stdout:131: finishing write tid 5 to smithi07929762-1 2024-04-24T01:55:45.891 INFO:tasks.rados.rados.0.smithi079.stdout:131: oid 1 updating version 0 to 45 2024-04-24T01:55:45.891 INFO:tasks.rados.rados.0.smithi079.stdout:131: oid 1 version 45 is already newer than 45 2024-04-24T01:55:45.891 INFO:tasks.rados.rados.0.smithi079.stdout:131: oid 1 version 45 is already newer than 45 2024-04-24T01:55:45.891 INFO:tasks.rados.rados.0.smithi079.stdout:131: oid 1 version 45 is already newer than 45 2024-04-24T01:55:45.891 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 1 v 45 (ObjNum 86 snap 6 seq_num 86) dirty exists 2024-04-24T01:55:45.892 INFO:tasks.rados.rados.0.smithi079.stdout:131: left oid 1 (ObjNum 86 snap 6 seq_num 86) 2024-04-24T01:55:45.892 INFO:tasks.rados.rados.0.smithi079.stdout:127: finishing write tid 3 to smithi07929762-35 2024-04-24T01:55:45.892 INFO:tasks.rados.rados.0.smithi079.stdout:127: finishing write tid 4 to smithi07929762-35 2024-04-24T01:55:45.892 INFO:tasks.rados.rados.0.smithi079.stdout:127: finishing write tid 5 to smithi07929762-35 2024-04-24T01:55:45.892 INFO:tasks.rados.rados.0.smithi079.stdout:127: oid 35 updating version 0 to 14 2024-04-24T01:55:45.892 INFO:tasks.rados.rados.0.smithi079.stdout:127: oid 35 updating version 14 to 17 2024-04-24T01:55:45.892 INFO:tasks.rados.rados.0.smithi079.stdout:127: oid 35 version 17 is already newer than 17 2024-04-24T01:55:45.892 INFO:tasks.rados.rados.0.smithi079.stdout:127: oid 35 version 17 is already newer than 17 2024-04-24T01:55:45.892 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 35 v 17 (ObjNum 84 snap 6 seq_num 84) dirty exists 2024-04-24T01:55:45.892 INFO:tasks.rados.rados.0.smithi079.stdout:127: left oid 35 (ObjNum 84 snap 6 seq_num 84) 2024-04-24T01:55:45.892 INFO:tasks.rados.rados.0.smithi079.stdout:128: finishing write tid 2 to smithi07929762-2 2024-04-24T01:55:45.892 INFO:tasks.rados.rados.0.smithi079.stdout:133: finishing write tid 4 to smithi07929762-43 2024-04-24T01:55:45.892 INFO:tasks.rados.rados.0.smithi079.stdout:133: finishing write tid 5 to smithi07929762-43 2024-04-24T01:55:45.892 INFO:tasks.rados.rados.0.smithi079.stdout:133: oid 43 updating version 0 to 17 2024-04-24T01:55:45.892 INFO:tasks.rados.rados.0.smithi079.stdout:133: oid 43 version 17 is already newer than 17 2024-04-24T01:55:45.893 INFO:tasks.rados.rados.0.smithi079.stdout:133: oid 43 version 17 is already newer than 17 2024-04-24T01:55:45.893 INFO:tasks.rados.rados.0.smithi079.stdout:133: oid 43 version 17 is already newer than 17 2024-04-24T01:55:45.893 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 43 v 17 (ObjNum 87 snap 6 seq_num 87) dirty exists 2024-04-24T01:55:45.893 INFO:tasks.rados.rados.0.smithi079.stdout:133: left oid 43 (ObjNum 87 snap 6 seq_num 87) 2024-04-24T01:55:45.893 INFO:tasks.rados.rados.0.smithi079.stdout:128: finishing write tid 3 to smithi07929762-2 2024-04-24T01:55:45.893 INFO:tasks.rados.rados.0.smithi079.stdout:128: finishing write tid 4 to smithi07929762-2 2024-04-24T01:55:45.893 INFO:tasks.rados.rados.0.smithi079.stdout:132: expect (ObjNum 18 snap 0 seq_num 18) 2024-04-24T01:55:45.975 INFO:tasks.rados.rados.0.smithi079.stdout:128: finishing write tid 5 to smithi07929762-2 2024-04-24T01:55:45.975 INFO:tasks.rados.rados.0.smithi079.stdout:128: oid 2 updating version 0 to 11 2024-04-24T01:55:45.976 INFO:tasks.rados.rados.0.smithi079.stdout:128: oid 2 version 11 is already newer than 11 2024-04-24T01:55:45.976 INFO:tasks.rados.rados.0.smithi079.stdout:128: oid 2 version 11 is already newer than 11 2024-04-24T01:55:45.976 INFO:tasks.rados.rados.0.smithi079.stdout:128: oid 2 version 11 is already newer than 11 2024-04-24T01:55:45.976 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 2 v 11 (ObjNum 85 snap 6 seq_num 85) dirty exists 2024-04-24T01:55:45.976 INFO:tasks.rados.rados.0.smithi079.stdout:128: left oid 2 (ObjNum 85 snap 6 seq_num 85) 2024-04-24T01:55:46.269 INFO:tasks.rados.rados.0.smithi079.stdout:116: done (15 left) 2024-04-24T01:55:46.269 INFO:tasks.rados.rados.0.smithi079.stdout:117: done (14 left) 2024-04-24T01:55:46.269 INFO:tasks.rados.rados.0.smithi079.stdout:118: done (13 left) 2024-04-24T01:55:46.270 INFO:tasks.rados.rados.0.smithi079.stdout:119: done (12 left) 2024-04-24T01:55:46.270 INFO:tasks.rados.rados.0.smithi079.stdout:120: done (11 left) 2024-04-24T01:55:46.270 INFO:tasks.rados.rados.0.smithi079.stdout:121: done (10 left) 2024-04-24T01:55:46.270 INFO:tasks.rados.rados.0.smithi079.stdout:123: done (9 left) 2024-04-24T01:55:46.270 INFO:tasks.rados.rados.0.smithi079.stdout:126: done (8 left) 2024-04-24T01:55:46.270 INFO:tasks.rados.rados.0.smithi079.stdout:127: done (7 left) 2024-04-24T01:55:46.270 INFO:tasks.rados.rados.0.smithi079.stdout:128: done (6 left) 2024-04-24T01:55:46.270 INFO:tasks.rados.rados.0.smithi079.stdout:129: done (5 left) 2024-04-24T01:55:46.270 INFO:tasks.rados.rados.0.smithi079.stdout:130: done (4 left) 2024-04-24T01:55:46.270 INFO:tasks.rados.rados.0.smithi079.stdout:131: done (3 left) 2024-04-24T01:55:46.270 INFO:tasks.rados.rados.0.smithi079.stdout:132: done (2 left) 2024-04-24T01:55:46.270 INFO:tasks.rados.rados.0.smithi079.stdout:133: done (1 left) 2024-04-24T01:55:46.270 INFO:tasks.rados.rados.0.smithi079.stdout:134: done (0 left) 2024-04-24T01:55:46.270 INFO:tasks.rados.rados.0.smithi079.stdout:135: write (excl) oid 42 current snap is 7 2024-04-24T01:55:46.271 INFO:tasks.rados.rados.0.smithi079.stdout:135: seq_num 88 ranges {679016=701795,1866533=496024,2900635=402643} 2024-04-24T01:55:46.277 INFO:tasks.rados.rados.0.smithi079.stdout:135: writing smithi07929762-42 from 679016 to 1380811 tid 1 2024-04-24T01:55:46.283 INFO:tasks.rados.rados.0.smithi079.stdout:135: writing smithi07929762-42 from 1866533 to 2362557 tid 2 2024-04-24T01:55:46.289 INFO:tasks.rados.rados.0.smithi079.stdout:135: writing smithi07929762-42 from 2900635 to 3303278 tid 3 2024-04-24T01:55:46.289 INFO:tasks.rados.rados.0.smithi079.stdout:136: snap_remove snap 6 2024-04-24T01:55:46.393 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T01:55:46.497 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:46.495+0000 7f90882ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:46.501 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:46.500+0000 7f90882ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:47.273 INFO:tasks.rados.rados.0.smithi079.stdout:136: done (1 left) 2024-04-24T01:55:47.273 INFO:tasks.rados.rados.0.smithi079.stdout:137: read oid 2 snap -1 2024-04-24T01:55:47.273 INFO:tasks.rados.rados.0.smithi079.stdout:137: expect (ObjNum 85 snap 6 seq_num 85) 2024-04-24T01:55:47.273 INFO:tasks.rados.rados.0.smithi079.stdout:138: read oid 16 snap -1 2024-04-24T01:55:47.273 INFO:tasks.rados.rados.0.smithi079.stdout:138: expect (ObjNum 15 snap 0 seq_num 15) 2024-04-24T01:55:47.273 INFO:tasks.rados.rados.0.smithi079.stdout:139: read oid 32 snap -1 2024-04-24T01:55:47.273 INFO:tasks.rados.rados.0.smithi079.stdout:139: expect (ObjNum 31 snap 0 seq_num 31) 2024-04-24T01:55:47.274 INFO:tasks.rados.rados.0.smithi079.stdout:140: read oid 22 snap -1 2024-04-24T01:55:47.274 INFO:tasks.rados.rados.0.smithi079.stdout:140: expect (ObjNum 68 snap 3 seq_num 68) 2024-04-24T01:55:47.274 INFO:tasks.rados.rados.0.smithi079.stdout:141: write oid 17 current snap is 7 2024-04-24T01:55:47.274 INFO:tasks.rados.rados.0.smithi079.stdout:135: finishing write tid 1 to smithi07929762-42 2024-04-24T01:55:47.274 INFO:tasks.rados.rados.0.smithi079.stdout:141: seq_num 89 ranges {695823=777044,1925015=639682,3328058=167765} 2024-04-24T01:55:47.281 INFO:tasks.rados.rados.0.smithi079.stdout:141: writing smithi07929762-17 from 695823 to 1472867 tid 1 2024-04-24T01:55:47.288 INFO:tasks.rados.rados.0.smithi079.stdout:141: writing smithi07929762-17 from 1925015 to 2564697 tid 2 2024-04-24T01:55:47.294 INFO:tasks.rados.rados.0.smithi079.stdout:141: writing smithi07929762-17 from 3328058 to 3495823 tid 3 2024-04-24T01:55:47.294 INFO:tasks.rados.rados.0.smithi079.stdout:142: write (excl) oid 36 current snap is 7 2024-04-24T01:55:47.294 INFO:tasks.rados.rados.0.smithi079.stdout:135: finishing write tid 2 to smithi07929762-42 2024-04-24T01:55:47.294 INFO:tasks.rados.rados.0.smithi079.stdout:135: finishing write tid 3 to smithi07929762-42 2024-04-24T01:55:47.294 INFO:tasks.rados.rados.0.smithi079.stdout:135: finishing write tid 4 to smithi07929762-42 2024-04-24T01:55:47.294 INFO:tasks.rados.rados.0.smithi079.stdout:135: finishing write tid 5 to smithi07929762-42 2024-04-24T01:55:47.294 INFO:tasks.rados.rados.0.smithi079.stdout:135: oid 42 updating version 0 to 30 2024-04-24T01:55:47.294 INFO:tasks.rados.rados.0.smithi079.stdout:135: oid 42 updating version 30 to 33 2024-04-24T01:55:47.294 INFO:tasks.rados.rados.0.smithi079.stdout:135: oid 42 version 33 is already newer than 33 2024-04-24T01:55:47.294 INFO:tasks.rados.rados.0.smithi079.stdout:135: oid 42 version 33 is already newer than 33 2024-04-24T01:55:47.295 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 42 v 33 (ObjNum 88 snap 7 seq_num 88) dirty exists 2024-04-24T01:55:47.295 INFO:tasks.rados.rados.0.smithi079.stdout:135: left oid 42 (ObjNum 88 snap 7 seq_num 88) 2024-04-24T01:55:47.295 INFO:tasks.rados.rados.0.smithi079.stdout:142: seq_num 90 ranges {712630=452293,1583497=499693,2555481=425187} 2024-04-24T01:55:47.302 INFO:tasks.rados.rados.0.smithi079.stdout:142: writing smithi07929762-36 from 712630 to 1164923 tid 1 2024-04-24T01:55:47.309 INFO:tasks.rados.rados.0.smithi079.stdout:142: writing smithi07929762-36 from 1583497 to 2083190 tid 2 2024-04-24T01:55:47.314 INFO:tasks.rados.rados.0.smithi079.stdout:142: writing smithi07929762-36 from 2555481 to 2980668 tid 3 2024-04-24T01:55:47.314 INFO:tasks.rados.rados.0.smithi079.stdout:135: done (6 left) 2024-04-24T01:55:47.314 INFO:tasks.rados.rados.0.smithi079.stdout:143: write (excl) oid 29 current snap is 7 2024-04-24T01:55:47.314 INFO:tasks.rados.rados.0.smithi079.stdout:143: seq_num 91 ranges {729437=527542,1925626=643351,3266551=262886} 2024-04-24T01:55:47.321 INFO:tasks.rados.rados.0.smithi079.stdout:143: writing smithi07929762-29 from 729437 to 1256979 tid 1 2024-04-24T01:55:47.329 INFO:tasks.rados.rados.0.smithi079.stdout:143: writing smithi07929762-29 from 1925626 to 2568977 tid 2 2024-04-24T01:55:47.336 INFO:tasks.rados.rados.0.smithi079.stdout:143: writing smithi07929762-29 from 3266551 to 3529437 tid 3 2024-04-24T01:55:47.336 INFO:tasks.rados.rados.0.smithi079.stdout:144: write oid 14 current snap is 7 2024-04-24T01:55:47.336 INFO:tasks.rados.rados.0.smithi079.stdout:144: seq_num 92 ranges {746244=719144,2100461=503362,3126680=419564} 2024-04-24T01:55:47.345 INFO:tasks.rados.rados.0.smithi079.stdout:144: writing smithi07929762-14 from 746244 to 1465388 tid 1 2024-04-24T01:55:47.354 INFO:tasks.rados.rados.0.smithi079.stdout:144: writing smithi07929762-14 from 2100461 to 2603823 tid 2 2024-04-24T01:55:47.361 INFO:tasks.rados.rados.0.smithi079.stdout:144: writing smithi07929762-14 from 3126680 to 3546244 tid 3 2024-04-24T01:55:47.361 INFO:tasks.rados.rados.0.smithi079.stdout:145: write oid 13 current snap is 7 2024-04-24T01:55:47.361 INFO:tasks.rados.rados.0.smithi079.stdout:145: seq_num 93 ranges {763051=794393,2158943=647020,3437750=125301} 2024-04-24T01:55:47.373 INFO:tasks.rados.rados.0.smithi079.stdout:145: writing smithi07929762-13 from 763051 to 1557444 tid 1 2024-04-24T01:55:47.382 INFO:tasks.rados.rados.0.smithi079.stdout:145: writing smithi07929762-13 from 2158943 to 2805963 tid 2 2024-04-24T01:55:47.387 INFO:tasks.rados.rados.0.smithi079.stdout:145: writing smithi07929762-13 from 3437750 to 3563051 tid 3 2024-04-24T01:55:47.387 INFO:tasks.rados.rados.0.smithi079.stdout:146: write oid 33 current snap is 7 2024-04-24T01:55:47.387 INFO:tasks.rados.rados.0.smithi079.stdout:146: seq_num 94 ranges {779858=469642,1817425=507031,2781526=586628} 2024-04-24T01:55:47.396 INFO:tasks.rados.rados.0.smithi079.stdout:146: writing smithi07929762-33 from 779858 to 1249500 tid 1 2024-04-24T01:55:47.404 INFO:tasks.rados.rados.0.smithi079.stdout:146: writing smithi07929762-33 from 1817425 to 2324456 tid 2 2024-04-24T01:55:47.413 INFO:tasks.rados.rados.0.smithi079.stdout:146: writing smithi07929762-33 from 2781526 to 3368154 tid 3 2024-04-24T01:55:47.413 INFO:tasks.rados.rados.0.smithi079.stdout:147: read oid 34 snap -1 2024-04-24T01:55:47.414 INFO:tasks.rados.rados.0.smithi079.stdout:147: expect (ObjNum 33 snap 0 seq_num 33) 2024-04-24T01:55:47.414 INFO:tasks.rados.rados.0.smithi079.stdout:148: read oid 47 snap -1 2024-04-24T01:55:47.415 INFO:tasks.rados.rados.0.smithi079.stdout:148: expect (ObjNum 46 snap 0 seq_num 46) 2024-04-24T01:55:47.415 INFO:tasks.rados.rados.0.smithi079.stdout:149: delete oid 49 current snap is 7 2024-04-24T01:55:47.415 INFO:tasks.rados.rados.0.smithi079.stdout:140: expect (ObjNum 68 snap 3 seq_num 68) 2024-04-24T01:55:47.603 INFO:tasks.rados.rados.0.smithi079.stdout:138: expect (ObjNum 15 snap 0 seq_num 15) 2024-04-24T01:55:47.688 INFO:tasks.rados.rados.0.smithi079.stdout:139: expect (ObjNum 31 snap 0 seq_num 31) 2024-04-24T01:55:47.780 INFO:tasks.rados.rados.0.smithi079.stdout:142: finishing write tid 1 to smithi07929762-36 2024-04-24T01:55:47.781 INFO:tasks.rados.rados.0.smithi079.stdout:137: expect (ObjNum 85 snap 6 seq_num 85) 2024-04-24T01:55:47.963 INFO:tasks.rados.rados.0.smithi079.stdout:141: finishing write tid 1 to smithi07929762-17 2024-04-24T01:55:47.994 INFO:tasks.rados.rados.0.smithi079.stdout:142: finishing write tid 2 to smithi07929762-36 2024-04-24T01:55:47.995 INFO:tasks.rados.rados.0.smithi079.stdout:141: finishing write tid 2 to smithi07929762-17 2024-04-24T01:55:47.995 INFO:tasks.rados.rados.0.smithi079.stdout:142: finishing write tid 3 to smithi07929762-36 2024-04-24T01:55:47.995 INFO:tasks.rados.rados.0.smithi079.stdout:137: done (12 left) 2024-04-24T01:55:47.995 INFO:tasks.rados.rados.0.smithi079.stdout:138: done (11 left) 2024-04-24T01:55:47.995 INFO:tasks.rados.rados.0.smithi079.stdout:139: done (10 left) 2024-04-24T01:55:47.995 INFO:tasks.rados.rados.0.smithi079.stdout:140: done (9 left) 2024-04-24T01:55:47.995 INFO:tasks.rados.rados.0.smithi079.stdout:149: done (8 left) 2024-04-24T01:55:47.995 INFO:tasks.rados.rados.0.smithi079.stdout:150: write (excl) oid 1 current snap is 7 2024-04-24T01:55:47.995 INFO:tasks.rados.rados.0.smithi079.stdout:142: finishing write tid 4 to smithi07929762-36 2024-04-24T01:55:47.995 INFO:tasks.rados.rados.0.smithi079.stdout:141: finishing write tid 3 to smithi07929762-17 2024-04-24T01:55:47.995 INFO:tasks.rados.rados.0.smithi079.stdout:142: finishing write tid 5 to smithi07929762-36 2024-04-24T01:55:47.995 INFO:tasks.rados.rados.0.smithi079.stdout:142: oid 36 updating version 0 to 19 2024-04-24T01:55:47.996 INFO:tasks.rados.rados.0.smithi079.stdout:142: oid 36 version 19 is already newer than 16 2024-04-24T01:55:47.996 INFO:tasks.rados.rados.0.smithi079.stdout:142: oid 36 version 19 is already newer than 19 2024-04-24T01:55:47.996 INFO:tasks.rados.rados.0.smithi079.stdout:142: oid 36 version 19 is already newer than 19 2024-04-24T01:55:47.996 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 36 v 19 (ObjNum 90 snap 7 seq_num 90) dirty exists 2024-04-24T01:55:47.996 INFO:tasks.rados.rados.0.smithi079.stdout:142: left oid 36 (ObjNum 90 snap 7 seq_num 90) 2024-04-24T01:55:47.996 INFO:tasks.rados.rados.0.smithi079.stdout:141: finishing write tid 4 to smithi07929762-17 2024-04-24T01:55:47.996 INFO:tasks.rados.rados.0.smithi079.stdout:141: finishing write tid 5 to smithi07929762-17 2024-04-24T01:55:47.996 INFO:tasks.rados.rados.0.smithi079.stdout:141: oid 17 updating version 0 to 29 2024-04-24T01:55:47.996 INFO:tasks.rados.rados.0.smithi079.stdout:141: oid 17 version 29 is already newer than 29 2024-04-24T01:55:47.996 INFO:tasks.rados.rados.0.smithi079.stdout:141: oid 17 version 29 is already newer than 29 2024-04-24T01:55:47.996 INFO:tasks.rados.rados.0.smithi079.stdout:141: oid 17 version 29 is already newer than 29 2024-04-24T01:55:47.996 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 17 v 29 (ObjNum 89 snap 7 seq_num 89) dirty exists 2024-04-24T01:55:47.996 INFO:tasks.rados.rados.0.smithi079.stdout:141: left oid 17 (ObjNum 89 snap 7 seq_num 89) 2024-04-24T01:55:47.996 INFO:tasks.rados.rados.0.smithi079.stdout:150: seq_num 96 ranges {413472=620140,1818036=510700,3120019=493453} 2024-04-24T01:55:47.997 INFO:tasks.rados.rados.0.smithi079.stdout:150: writing smithi07929762-1 from 413472 to 1033612 tid 1 2024-04-24T01:55:47.997 INFO:tasks.rados.rados.0.smithi079.stdout:150: writing smithi07929762-1 from 1818036 to 2328736 tid 2 2024-04-24T01:55:47.998 INFO:tasks.rados.rados.0.smithi079.stdout:150: writing smithi07929762-1 from 3120019 to 3613472 tid 3 2024-04-24T01:55:47.998 INFO:tasks.rados.rados.0.smithi079.stdout:141: done (8 left) 2024-04-24T01:55:47.998 INFO:tasks.rados.rados.0.smithi079.stdout:142: done (7 left) 2024-04-24T01:55:47.998 INFO:tasks.rados.rados.0.smithi079.stdout:151: delete oid 45 current snap is 7 2024-04-24T01:55:47.998 INFO:tasks.rados.rados.0.smithi079.stdout:143: finishing write tid 1 to smithi07929762-29 2024-04-24T01:55:47.998 INFO:tasks.rados.rados.0.smithi079.stdout:147: expect (ObjNum 33 snap 0 seq_num 33) 2024-04-24T01:55:48.090 INFO:tasks.rados.rados.0.smithi079.stdout:146: finishing write tid 1 to smithi07929762-33 2024-04-24T01:55:48.090 INFO:tasks.rados.rados.0.smithi079.stdout:145: finishing write tid 1 to smithi07929762-13 2024-04-24T01:55:48.090 INFO:tasks.rados.rados.0.smithi079.stdout:146: finishing write tid 2 to smithi07929762-33 2024-04-24T01:55:48.090 INFO:tasks.rados.rados.0.smithi079.stdout:145: finishing write tid 2 to smithi07929762-13 2024-04-24T01:55:48.090 INFO:tasks.rados.rados.0.smithi079.stdout:145: finishing write tid 3 to smithi07929762-13 2024-04-24T01:55:48.091 INFO:tasks.rados.rados.0.smithi079.stdout:143: finishing write tid 2 to smithi07929762-29 2024-04-24T01:55:48.091 INFO:tasks.rados.rados.0.smithi079.stdout:146: finishing write tid 3 to smithi07929762-33 2024-04-24T01:55:48.091 INFO:tasks.rados.rados.0.smithi079.stdout:146: finishing write tid 4 to smithi07929762-33 2024-04-24T01:55:48.091 INFO:tasks.rados.rados.0.smithi079.stdout:143: finishing write tid 3 to smithi07929762-29 2024-04-24T01:55:48.091 INFO:tasks.rados.rados.0.smithi079.stdout:146: finishing write tid 5 to smithi07929762-33 2024-04-24T01:55:48.091 INFO:tasks.rados.rados.0.smithi079.stdout:146: oid 33 updating version 0 to 24 2024-04-24T01:55:48.092 INFO:tasks.rados.rados.0.smithi079.stdout:146: oid 33 version 24 is already newer than 24 2024-04-24T01:55:48.092 INFO:tasks.rados.rados.0.smithi079.stdout:146: oid 33 version 24 is already newer than 24 2024-04-24T01:55:48.092 INFO:tasks.rados.rados.0.smithi079.stdout:146: oid 33 version 24 is already newer than 24 2024-04-24T01:55:48.092 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 33 v 24 (ObjNum 94 snap 7 seq_num 94) dirty exists 2024-04-24T01:55:48.092 INFO:tasks.rados.rados.0.smithi079.stdout:146: left oid 33 (ObjNum 94 snap 7 seq_num 94) 2024-04-24T01:55:48.092 INFO:tasks.rados.rados.0.smithi079.stdout:143: finishing write tid 4 to smithi07929762-29 2024-04-24T01:55:48.093 INFO:tasks.rados.rados.0.smithi079.stdout:143: finishing write tid 5 to smithi07929762-29 2024-04-24T01:55:48.093 INFO:tasks.rados.rados.0.smithi079.stdout:143: oid 29 updating version 0 to 34 2024-04-24T01:55:48.093 INFO:tasks.rados.rados.0.smithi079.stdout:143: oid 29 updating version 34 to 37 2024-04-24T01:55:48.093 INFO:tasks.rados.rados.0.smithi079.stdout:143: oid 29 version 37 is already newer than 37 2024-04-24T01:55:48.093 INFO:tasks.rados.rados.0.smithi079.stdout:143: oid 29 version 37 is already newer than 37 2024-04-24T01:55:48.093 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 29 v 37 (ObjNum 91 snap 7 seq_num 91) dirty exists 2024-04-24T01:55:48.093 INFO:tasks.rados.rados.0.smithi079.stdout:143: left oid 29 (ObjNum 91 snap 7 seq_num 91) 2024-04-24T01:55:48.094 INFO:tasks.rados.rados.0.smithi079.stdout:145: finishing write tid 4 to smithi07929762-13 2024-04-24T01:55:48.094 INFO:tasks.rados.rados.0.smithi079.stdout:143: done (7 left) 2024-04-24T01:55:48.094 INFO:tasks.rados.rados.0.smithi079.stdout:146: done (6 left) 2024-04-24T01:55:48.094 INFO:tasks.rados.rados.0.smithi079.stdout:147: done (5 left) 2024-04-24T01:55:48.094 INFO:tasks.rados.rados.0.smithi079.stdout:151: done (4 left) 2024-04-24T01:55:48.094 INFO:tasks.rados.rados.0.smithi079.stdout:152: write (excl) oid 5 current snap is 7 2024-04-24T01:55:48.094 INFO:tasks.rados.rados.0.smithi079.stdout:144: finishing write tid 1 to smithi07929762-14 2024-04-24T01:55:48.095 INFO:tasks.rados.rados.0.smithi079.stdout:152: seq_num 98 ranges {447086=770638,1935000=798016,3458512=188574} 2024-04-24T01:55:48.095 INFO:tasks.rados.rados.0.smithi079.stdout:152: writing smithi07929762-5 from 447086 to 1217724 tid 1 2024-04-24T01:55:48.106 INFO:tasks.rados.rados.0.smithi079.stdout:152: writing smithi07929762-5 from 1935000 to 2733016 tid 2 2024-04-24T01:55:48.111 INFO:tasks.rados.rados.0.smithi079.stdout:152: writing smithi07929762-5 from 3458512 to 3647086 tid 3 2024-04-24T01:55:48.112 INFO:tasks.rados.rados.0.smithi079.stdout:153: read oid 50 snap -1 2024-04-24T01:55:48.112 INFO:tasks.rados.rados.0.smithi079.stdout:153: expect (ObjNum 79 snap 6 seq_num 79) 2024-04-24T01:55:48.112 INFO:tasks.rados.rados.0.smithi079.stdout:154: snap_create 2024-04-24T01:55:48.112 INFO:tasks.rados.rados.0.smithi079.stdout:145: finishing write tid 5 to smithi07929762-13 2024-04-24T01:55:48.112 INFO:tasks.rados.rados.0.smithi079.stdout:145: oid 13 updating version 0 to 28 2024-04-24T01:55:48.112 INFO:tasks.rados.rados.0.smithi079.stdout:145: oid 13 version 28 is already newer than 28 2024-04-24T01:55:48.112 INFO:tasks.rados.rados.0.smithi079.stdout:145: oid 13 version 28 is already newer than 28 2024-04-24T01:55:48.112 INFO:tasks.rados.rados.0.smithi079.stdout:145: oid 13 version 28 is already newer than 28 2024-04-24T01:55:48.112 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 13 v 28 (ObjNum 93 snap 7 seq_num 93) dirty exists 2024-04-24T01:55:48.112 INFO:tasks.rados.rados.0.smithi079.stdout:145: left oid 13 (ObjNum 93 snap 7 seq_num 93) 2024-04-24T01:55:48.112 INFO:tasks.rados.rados.0.smithi079.stdout:148: expect (ObjNum 46 snap 0 seq_num 46) 2024-04-24T01:55:48.219 INFO:tasks.rados.rados.0.smithi079.stdout:144: finishing write tid 2 to smithi07929762-14 2024-04-24T01:55:48.220 INFO:tasks.rados.rados.0.smithi079.stdout:144: finishing write tid 3 to smithi07929762-14 2024-04-24T01:55:48.220 INFO:tasks.rados.rados.0.smithi079.stdout:144: finishing write tid 4 to smithi07929762-14 2024-04-24T01:55:48.220 INFO:tasks.rados.rados.0.smithi079.stdout:144: finishing write tid 5 to smithi07929762-14 2024-04-24T01:55:48.220 INFO:tasks.rados.rados.0.smithi079.stdout:144: oid 14 updating version 0 to 41 2024-04-24T01:55:48.220 INFO:tasks.rados.rados.0.smithi079.stdout:144: oid 14 version 41 is already newer than 41 2024-04-24T01:55:48.220 INFO:tasks.rados.rados.0.smithi079.stdout:144: oid 14 version 41 is already newer than 41 2024-04-24T01:55:48.220 INFO:tasks.rados.rados.0.smithi079.stdout:144: oid 14 version 41 is already newer than 41 2024-04-24T01:55:48.220 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 14 v 41 (ObjNum 92 snap 7 seq_num 92) dirty exists 2024-04-24T01:55:48.220 INFO:tasks.rados.rados.0.smithi079.stdout:144: left oid 14 (ObjNum 92 snap 7 seq_num 92) 2024-04-24T01:55:48.220 INFO:tasks.rados.rados.0.smithi079.stdout:150: finishing write tid 1 to smithi07929762-1 2024-04-24T01:55:48.220 INFO:tasks.rados.rados.0.smithi079.stdout:150: finishing write tid 2 to smithi07929762-1 2024-04-24T01:55:48.220 INFO:tasks.rados.rados.0.smithi079.stdout:150: finishing write tid 3 to smithi07929762-1 2024-04-24T01:55:48.221 INFO:tasks.rados.rados.0.smithi079.stdout:150: finishing write tid 4 to smithi07929762-1 2024-04-24T01:55:48.221 INFO:tasks.rados.rados.0.smithi079.stdout:150: finishing write tid 5 to smithi07929762-1 2024-04-24T01:55:48.221 INFO:tasks.rados.rados.0.smithi079.stdout:150: oid 1 updating version 0 to 50 2024-04-24T01:55:48.221 INFO:tasks.rados.rados.0.smithi079.stdout:150: oid 1 version 50 is already newer than 47 2024-04-24T01:55:48.221 INFO:tasks.rados.rados.0.smithi079.stdout:150: oid 1 version 50 is already newer than 50 2024-04-24T01:55:48.221 INFO:tasks.rados.rados.0.smithi079.stdout:150: oid 1 version 50 is already newer than 50 2024-04-24T01:55:48.221 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 1 v 50 (ObjNum 96 snap 7 seq_num 96) dirty exists 2024-04-24T01:55:48.221 INFO:tasks.rados.rados.0.smithi079.stdout:150: left oid 1 (ObjNum 96 snap 7 seq_num 96) 2024-04-24T01:55:48.221 INFO:tasks.rados.rados.0.smithi079.stdout:152: finishing write tid 1 to smithi07929762-5 2024-04-24T01:55:48.221 INFO:tasks.rados.rados.0.smithi079.stdout:152: finishing write tid 2 to smithi07929762-5 2024-04-24T01:55:48.221 INFO:tasks.rados.rados.0.smithi079.stdout:152: finishing write tid 3 to smithi07929762-5 2024-04-24T01:55:48.221 INFO:tasks.rados.rados.0.smithi079.stdout:152: finishing write tid 4 to smithi07929762-5 2024-04-24T01:55:48.221 INFO:tasks.rados.rados.0.smithi079.stdout:152: finishing write tid 5 to smithi07929762-5 2024-04-24T01:55:48.221 INFO:tasks.rados.rados.0.smithi079.stdout:152: oid 5 updating version 0 to 38 2024-04-24T01:55:48.221 INFO:tasks.rados.rados.0.smithi079.stdout:152: oid 5 updating version 38 to 41 2024-04-24T01:55:48.222 INFO:tasks.rados.rados.0.smithi079.stdout:152: oid 5 version 41 is already newer than 41 2024-04-24T01:55:48.222 INFO:tasks.rados.rados.0.smithi079.stdout:152: oid 5 version 41 is already newer than 41 2024-04-24T01:55:48.222 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 5 v 41 (ObjNum 98 snap 7 seq_num 98) dirty exists 2024-04-24T01:55:48.222 INFO:tasks.rados.rados.0.smithi079.stdout:152: left oid 5 (ObjNum 98 snap 7 seq_num 98) 2024-04-24T01:55:48.222 INFO:tasks.rados.rados.0.smithi079.stdout:153: expect (ObjNum 79 snap 6 seq_num 79) 2024-04-24T01:55:48.386 INFO:tasks.rados.rados.0.smithi079.stdout:144: done (6 left) 2024-04-24T01:55:48.386 INFO:tasks.rados.rados.0.smithi079.stdout:145: done (5 left) 2024-04-24T01:55:48.387 INFO:tasks.rados.rados.0.smithi079.stdout:148: done (4 left) 2024-04-24T01:55:48.387 INFO:tasks.rados.rados.0.smithi079.stdout:150: done (3 left) 2024-04-24T01:55:48.387 INFO:tasks.rados.rados.0.smithi079.stdout:152: done (2 left) 2024-04-24T01:55:48.387 INFO:tasks.rados.rados.0.smithi079.stdout:153: done (1 left) 2024-04-24T01:55:48.387 INFO:tasks.rados.rados.0.smithi079.stdout:154: done (0 left) 2024-04-24T01:55:48.387 INFO:tasks.rados.rados.0.smithi079.stdout:155: read oid 33 snap -1 2024-04-24T01:55:48.387 INFO:tasks.rados.rados.0.smithi079.stdout:155: expect (ObjNum 94 snap 7 seq_num 94) 2024-04-24T01:55:48.387 INFO:tasks.rados.rados.0.smithi079.stdout:156: write (excl) oid 26 current snap is 8 2024-04-24T01:55:48.388 INFO:tasks.rados.rados.0.smithi079.stdout:156: seq_num 99 ranges {463893=562240,1593482=658027,2685935=559341} 2024-04-24T01:55:48.392 INFO:tasks.rados.rados.0.smithi079.stdout:156: writing smithi07929762-26 from 463893 to 1026133 tid 1 2024-04-24T01:55:48.399 INFO:tasks.rados.rados.0.smithi079.stdout:156: writing smithi07929762-26 from 1593482 to 2251509 tid 2 2024-04-24T01:55:48.405 INFO:tasks.rados.rados.0.smithi079.stdout:156: writing smithi07929762-26 from 2685935 to 3245276 tid 3 2024-04-24T01:55:48.405 INFO:tasks.rados.rados.0.smithi079.stdout:157: delete oid 5 current snap is 8 2024-04-24T01:55:48.454 INFO:tasks.rados.rados.0.smithi079.stdout:155: expect (ObjNum 94 snap 7 seq_num 94) 2024-04-24T01:55:48.596 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T01:55:48.600 INFO:tasks.rados.rados.0.smithi079.stdout:156: finishing write tid 1 to smithi07929762-26 2024-04-24T01:55:48.600 INFO:tasks.rados.rados.0.smithi079.stdout:156: finishing write tid 2 to smithi07929762-26 2024-04-24T01:55:48.600 INFO:tasks.rados.rados.0.smithi079.stdout:156: finishing write tid 3 to smithi07929762-26 2024-04-24T01:55:48.600 INFO:tasks.rados.rados.0.smithi079.stdout:156: finishing write tid 4 to smithi07929762-26 2024-04-24T01:55:48.600 INFO:tasks.rados.rados.0.smithi079.stdout:156: finishing write tid 5 to smithi07929762-26 2024-04-24T01:55:48.600 INFO:tasks.rados.rados.0.smithi079.stdout:156: oid 26 updating version 0 to 30 2024-04-24T01:55:48.600 INFO:tasks.rados.rados.0.smithi079.stdout:156: oid 26 updating version 30 to 33 2024-04-24T01:55:48.600 INFO:tasks.rados.rados.0.smithi079.stdout:156: oid 26 version 33 is already newer than 33 2024-04-24T01:55:48.600 INFO:tasks.rados.rados.0.smithi079.stdout:156: oid 26 version 33 is already newer than 32 2024-04-24T01:55:48.601 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 26 v 33 (ObjNum 99 snap 8 seq_num 99) dirty exists 2024-04-24T01:55:48.601 INFO:tasks.rados.rados.0.smithi079.stdout:156: left oid 26 (ObjNum 99 snap 8 seq_num 99) 2024-04-24T01:55:48.601 INFO:tasks.rados.rados.0.smithi079.stdout:155: done (2 left) 2024-04-24T01:55:48.601 INFO:tasks.rados.rados.0.smithi079.stdout:156: done (1 left) 2024-04-24T01:55:48.601 INFO:tasks.rados.rados.0.smithi079.stdout:157: done (0 left) 2024-04-24T01:55:48.601 INFO:tasks.rados.rados.0.smithi079.stdout:158: read oid 45 snap 7 2024-04-24T01:55:48.601 INFO:tasks.rados.rados.0.smithi079.stdout:158: expect deleted 2024-04-24T01:55:48.601 INFO:tasks.rados.rados.0.smithi079.stdout:159: read oid 31 snap -1 2024-04-24T01:55:48.601 INFO:tasks.rados.rados.0.smithi079.stdout:159: expect (ObjNum 30 snap 0 seq_num 30) 2024-04-24T01:55:48.601 INFO:tasks.rados.rados.0.smithi079.stdout:160: snap_create 2024-04-24T01:55:48.622 INFO:tasks.rados.rados.0.smithi079.stdout:159: expect (ObjNum 30 snap 0 seq_num 30) 2024-04-24T01:55:48.703 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:48.701+0000 7f626e354640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:48.707 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:48.706+0000 7f626e354640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:49.278 INFO:tasks.rados.rados.0.smithi079.stdout:158: done (2 left) 2024-04-24T01:55:49.278 INFO:tasks.rados.rados.0.smithi079.stdout:159: done (1 left) 2024-04-24T01:55:49.278 INFO:tasks.rados.rados.0.smithi079.stdout:160: done (0 left) 2024-04-24T01:55:49.278 INFO:tasks.rados.rados.0.smithi079.stdout:161: snap_remove snap 8 2024-04-24T01:55:49.564 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-04-24T01:55:49.564 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-04-24T01:55:49.564 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-04-24T01:55:49.564 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-04-24T01:55:49.675 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:49.670+0000 7f26a6ced640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:49.679 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:49.679+0000 7f26a6ced640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:49.695 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T01:55:49.695 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T01:55:49.695 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T01:55:49.695 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T01:55:49.695 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T01:55:49.695 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T01:55:49.696 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T01:55:49.696 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T01:55:49.696 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T01:55:49.696 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T01:55:49.696 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T01:55:49.696 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T01:55:49.698 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T01:55:49.699 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T01:55:49.699 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-04-24T01:55:49.806 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:49.800+0000 7f1948e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:49.810 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:49.809+0000 7f1948e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:49.825 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T01:55:49.825 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T01:55:49.826 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T01:55:49.826 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T01:55:49.826 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T01:55:49.826 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T01:55:49.826 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T01:55:49.826 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T01:55:49.826 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T01:55:49.826 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T01:55:49.826 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T01:55:49.826 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T01:55:49.828 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T01:55:49.829 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T01:55:49.829 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-04-24T01:55:49.937 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:49.932+0000 7f6f4a15d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:49.941 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:49.940+0000 7f6f4a15d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:49.957 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T01:55:49.958 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T01:55:49.958 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T01:55:49.958 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T01:55:49.958 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T01:55:49.958 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T01:55:49.958 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T01:55:49.958 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T01:55:49.958 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T01:55:49.958 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T01:55:49.958 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T01:55:49.958 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T01:55:49.960 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T01:55:49.961 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T01:55:49.961 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-04-24T01:55:50.068 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:50.063+0000 7f0157f5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:50.072 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:50.071+0000 7f0157f5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:50.089 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T01:55:50.089 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T01:55:50.089 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T01:55:50.089 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T01:55:50.089 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T01:55:50.089 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T01:55:50.089 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T01:55:50.089 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T01:55:50.089 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T01:55:50.089 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T01:55:50.089 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T01:55:50.090 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T01:55:50.091 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T01:55:50.092 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T01:55:50.279 INFO:tasks.rados.rados.0.smithi079.stdout:161: done (0 left) 2024-04-24T01:55:50.279 INFO:tasks.rados.rados.0.smithi079.stdout:162: write (excl) oid 26 current snap is 9 2024-04-24T01:55:50.279 INFO:tasks.rados.rados.0.smithi079.stdout:162: seq_num 101 ranges {497507=712738,1710446=661696,3140781=556726} 2024-04-24T01:55:50.285 INFO:tasks.rados.rados.0.smithi079.stdout:162: writing smithi07929762-26 from 497507 to 1210245 tid 1 2024-04-24T01:55:50.291 INFO:tasks.rados.rados.0.smithi079.stdout:162: writing smithi07929762-26 from 1710446 to 2372142 tid 2 2024-04-24T01:55:50.300 INFO:tasks.rados.rados.0.smithi079.stdout:162: writing smithi07929762-26 from 3140781 to 3697507 tid 3 2024-04-24T01:55:50.300 INFO:tasks.rados.rados.0.smithi079.stdout:163: snap_remove snap 7 2024-04-24T01:55:50.801 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T01:55:50.906 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:50.904+0000 7f172246d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:50.910 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:50.909+0000 7f172246d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:51.285 INFO:tasks.rados.rados.0.smithi079.stdout:163: done (1 left) 2024-04-24T01:55:51.285 INFO:tasks.rados.rados.0.smithi079.stdout:164: read oid 7 snap -1 2024-04-24T01:55:51.285 INFO:tasks.rados.rados.0.smithi079.stdout:164: expect (ObjNum 67 snap 3 seq_num 67) 2024-04-24T01:55:51.285 INFO:tasks.rados.rados.0.smithi079.stdout:165: delete oid 12 current snap is 9 2024-04-24T01:55:51.285 INFO:tasks.rados.rados.0.smithi079.stdout:162: finishing write tid 1 to smithi07929762-26 2024-04-24T01:55:51.285 INFO:tasks.rados.rados.0.smithi079.stdout:162: finishing write tid 2 to smithi07929762-26 2024-04-24T01:55:51.285 INFO:tasks.rados.rados.0.smithi079.stdout:162: finishing write tid 3 to smithi07929762-26 2024-04-24T01:55:51.285 INFO:tasks.rados.rados.0.smithi079.stdout:162: finishing write tid 4 to smithi07929762-26 2024-04-24T01:55:51.285 INFO:tasks.rados.rados.0.smithi079.stdout:162: finishing write tid 5 to smithi07929762-26 2024-04-24T01:55:51.285 INFO:tasks.rados.rados.0.smithi079.stdout:162: oid 26 updating version 0 to 37 2024-04-24T01:55:51.285 INFO:tasks.rados.rados.0.smithi079.stdout:162: oid 26 version 37 is already newer than 37 2024-04-24T01:55:51.286 INFO:tasks.rados.rados.0.smithi079.stdout:162: oid 26 version 37 is already newer than 37 2024-04-24T01:55:51.286 INFO:tasks.rados.rados.0.smithi079.stdout:162: oid 26 version 37 is already newer than 34 2024-04-24T01:55:51.286 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 26 v 37 (ObjNum 101 snap 9 seq_num 101) dirty exists 2024-04-24T01:55:51.286 INFO:tasks.rados.rados.0.smithi079.stdout:162: left oid 26 (ObjNum 101 snap 9 seq_num 101) 2024-04-24T01:55:51.363 INFO:tasks.rados.rados.0.smithi079.stdout:162: done (2 left) 2024-04-24T01:55:51.364 INFO:tasks.rados.rados.0.smithi079.stdout:165: done (1 left) 2024-04-24T01:55:51.364 INFO:tasks.rados.rados.0.smithi079.stdout:166: delete oid 10 current snap is 9 2024-04-24T01:55:51.368 INFO:tasks.rados.rados.0.smithi079.stdout:164: expect (ObjNum 67 snap 3 seq_num 67) 2024-04-24T01:55:51.526 INFO:tasks.rados.rados.0.smithi079.stdout:164: done (1 left) 2024-04-24T01:55:51.526 INFO:tasks.rados.rados.0.smithi079.stdout:166: done (0 left) 2024-04-24T01:55:51.526 INFO:tasks.rados.rados.0.smithi079.stdout:167: delete oid 11 current snap is 9 2024-04-24T01:55:51.571 INFO:tasks.rados.rados.0.smithi079.stdout:167: done (0 left) 2024-04-24T01:55:51.572 INFO:tasks.rados.rados.0.smithi079.stdout:168: write (excl) oid 44 current snap is 9 2024-04-24T01:55:51.572 INFO:tasks.rados.rados.0.smithi079.stdout:168: seq_num 105 ranges {564735=613734,1828021=669034,3134120=630615} 2024-04-24T01:55:51.578 INFO:tasks.rados.rados.0.smithi079.stdout:168: writing smithi07929762-44 from 564735 to 1178469 tid 1 2024-04-24T01:55:51.585 INFO:tasks.rados.rados.0.smithi079.stdout:168: writing smithi07929762-44 from 1828021 to 2497055 tid 2 2024-04-24T01:55:51.593 INFO:tasks.rados.rados.0.smithi079.stdout:168: writing smithi07929762-44 from 3134120 to 3764735 tid 3 2024-04-24T01:55:51.593 INFO:tasks.rados.rados.0.smithi079.stdout:169: write oid 2 current snap is 9 2024-04-24T01:55:51.593 INFO:tasks.rados.rados.0.smithi079.stdout:169: seq_num 106 ranges {581542=688983,1886503=412692,2761543=670951} 2024-04-24T01:55:51.600 INFO:tasks.rados.rados.0.smithi079.stdout:169: writing smithi07929762-2 from 581542 to 1270525 tid 1 2024-04-24T01:55:51.605 INFO:tasks.rados.rados.0.smithi079.stdout:169: writing smithi07929762-2 from 1886503 to 2299195 tid 2 2024-04-24T01:55:51.614 INFO:tasks.rados.rados.0.smithi079.stdout:169: writing smithi07929762-2 from 2761543 to 3432494 tid 3 2024-04-24T01:55:51.614 INFO:tasks.rados.rados.0.smithi079.stdout:170: read oid 21 snap -1 2024-04-24T01:55:51.614 INFO:tasks.rados.rados.0.smithi079.stdout:170: expect (ObjNum 20 snap 0 seq_num 20) 2024-04-24T01:55:51.614 INFO:tasks.rados.rados.0.smithi079.stdout:171: write oid 17 current snap is 9 2024-04-24T01:55:51.614 INFO:tasks.rados.rados.0.smithi079.stdout:168: finishing write tid 1 to smithi07929762-44 2024-04-24T01:55:51.614 INFO:tasks.rados.rados.0.smithi079.stdout:171: seq_num 107 ranges {598349=480585,1544985=672703,2905319=482223} 2024-04-24T01:55:51.619 INFO:tasks.rados.rados.0.smithi079.stdout:171: writing smithi07929762-17 from 598349 to 1078934 tid 1 2024-04-24T01:55:51.626 INFO:tasks.rados.rados.0.smithi079.stdout:171: writing smithi07929762-17 from 1544985 to 2217688 tid 2 2024-04-24T01:55:51.633 INFO:tasks.rados.rados.0.smithi079.stdout:171: writing smithi07929762-17 from 2905319 to 3387542 tid 3 2024-04-24T01:55:51.633 INFO:tasks.rados.rados.0.smithi079.stdout:172: write (excl) oid 15 current snap is 9 2024-04-24T01:55:51.633 INFO:tasks.rados.rados.0.smithi079.stdout:172: seq_num 108 ranges {615156=555834,1603467=416361,2816389=693495} 2024-04-24T01:55:51.639 INFO:tasks.rados.rados.0.smithi079.stdout:172: writing smithi07929762-15 from 615156 to 1170990 tid 1 2024-04-24T01:55:51.645 INFO:tasks.rados.rados.0.smithi079.stdout:172: writing smithi07929762-15 from 1603467 to 2019828 tid 2 2024-04-24T01:55:51.655 INFO:tasks.rados.rados.0.smithi079.stdout:172: writing smithi07929762-15 from 2816389 to 3509884 tid 3 2024-04-24T01:55:51.655 INFO:tasks.rados.rados.0.smithi079.stdout:173: read oid 9 snap -1 2024-04-24T01:55:51.655 INFO:tasks.rados.rados.0.smithi079.stdout:173: expect (ObjNum 8 snap 0 seq_num 8) 2024-04-24T01:55:51.655 INFO:tasks.rados.rados.0.smithi079.stdout:174: write (excl) oid 39 current snap is 9 2024-04-24T01:55:51.655 INFO:tasks.rados.rados.0.smithi079.stdout:174: seq_num 109 ranges {631963=631083,2061949=560019,3243812=504767} 2024-04-24T01:55:51.662 INFO:tasks.rados.rados.0.smithi079.stdout:174: writing smithi07929762-39 from 631963 to 1263046 tid 1 2024-04-24T01:55:51.672 INFO:tasks.rados.rados.0.smithi079.stdout:174: writing smithi07929762-39 from 2061949 to 2621968 tid 2 2024-04-24T01:55:51.680 INFO:tasks.rados.rados.0.smithi079.stdout:174: writing smithi07929762-39 from 3243812 to 3748579 tid 3 2024-04-24T01:55:51.680 INFO:tasks.rados.rados.0.smithi079.stdout:175: read oid 3 snap -1 2024-04-24T01:55:51.680 INFO:tasks.rados.rados.0.smithi079.stdout:175: expect (ObjNum 70 snap 3 seq_num 70) 2024-04-24T01:55:51.681 INFO:tasks.rados.rados.0.smithi079.stdout:176: write oid 35 current snap is 9 2024-04-24T01:55:51.681 INFO:tasks.rados.rados.0.smithi079.stdout:176: seq_num 110 ranges {648770=706332,2120431=420030,3271235=432392} 2024-04-24T01:55:51.688 INFO:tasks.rados.rados.0.smithi079.stdout:176: writing smithi07929762-35 from 648770 to 1355102 tid 1 2024-04-24T01:55:51.694 INFO:tasks.rados.rados.0.smithi079.stdout:176: writing smithi07929762-35 from 2120431 to 2540461 tid 2 2024-04-24T01:55:51.701 INFO:tasks.rados.rados.0.smithi079.stdout:176: writing smithi07929762-35 from 3271235 to 3703627 tid 3 2024-04-24T01:55:51.701 INFO:tasks.rados.rados.0.smithi079.stdout:177: snap_create 2024-04-24T01:55:51.701 INFO:tasks.rados.rados.0.smithi079.stdout:169: finishing write tid 1 to smithi07929762-2 2024-04-24T01:55:51.701 INFO:tasks.rados.rados.0.smithi079.stdout:168: finishing write tid 2 to smithi07929762-44 2024-04-24T01:55:51.702 INFO:tasks.rados.rados.0.smithi079.stdout:170: expect (ObjNum 20 snap 0 seq_num 20) 2024-04-24T01:55:51.803 INFO:tasks.rados.rados.0.smithi079.stdout:173: expect (ObjNum 8 snap 0 seq_num 8) 2024-04-24T01:55:51.889 INFO:tasks.rados.rados.0.smithi079.stdout:168: finishing write tid 3 to smithi07929762-44 2024-04-24T01:55:51.889 INFO:tasks.rados.rados.0.smithi079.stdout:168: finishing write tid 4 to smithi07929762-44 2024-04-24T01:55:51.889 INFO:tasks.rados.rados.0.smithi079.stdout:169: finishing write tid 2 to smithi07929762-2 2024-04-24T01:55:51.889 INFO:tasks.rados.rados.0.smithi079.stdout:168: finishing write tid 5 to smithi07929762-44 2024-04-24T01:55:51.889 INFO:tasks.rados.rados.0.smithi079.stdout:168: oid 44 updating version 0 to 34 2024-04-24T01:55:51.889 INFO:tasks.rados.rados.0.smithi079.stdout:168: oid 44 updating version 34 to 37 2024-04-24T01:55:51.889 INFO:tasks.rados.rados.0.smithi079.stdout:168: oid 44 version 37 is already newer than 37 2024-04-24T01:55:51.889 INFO:tasks.rados.rados.0.smithi079.stdout:168: oid 44 version 37 is already newer than 37 2024-04-24T01:55:51.890 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 44 v 37 (ObjNum 105 snap 9 seq_num 105) dirty exists 2024-04-24T01:55:51.890 INFO:tasks.rados.rados.0.smithi079.stdout:168: left oid 44 (ObjNum 105 snap 9 seq_num 105) 2024-04-24T01:55:51.890 INFO:tasks.rados.rados.0.smithi079.stdout:169: finishing write tid 3 to smithi07929762-2 2024-04-24T01:55:51.890 INFO:tasks.rados.rados.0.smithi079.stdout:171: finishing write tid 1 to smithi07929762-17 2024-04-24T01:55:51.890 INFO:tasks.rados.rados.0.smithi079.stdout:169: finishing write tid 4 to smithi07929762-2 2024-04-24T01:55:51.890 INFO:tasks.rados.rados.0.smithi079.stdout:169: finishing write tid 5 to smithi07929762-2 2024-04-24T01:55:51.890 INFO:tasks.rados.rados.0.smithi079.stdout:169: oid 2 updating version 0 to 15 2024-04-24T01:55:51.890 INFO:tasks.rados.rados.0.smithi079.stdout:169: oid 2 version 15 is already newer than 15 2024-04-24T01:55:51.890 INFO:tasks.rados.rados.0.smithi079.stdout:169: oid 2 version 15 is already newer than 15 2024-04-24T01:55:51.890 INFO:tasks.rados.rados.0.smithi079.stdout:169: oid 2 version 15 is already newer than 15 2024-04-24T01:55:51.890 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 2 v 15 (ObjNum 106 snap 9 seq_num 106) dirty exists 2024-04-24T01:55:51.890 INFO:tasks.rados.rados.0.smithi079.stdout:169: left oid 2 (ObjNum 106 snap 9 seq_num 106) 2024-04-24T01:55:51.891 INFO:tasks.rados.rados.0.smithi079.stdout:171: finishing write tid 2 to smithi07929762-17 2024-04-24T01:55:51.891 INFO:tasks.rados.rados.0.smithi079.stdout:171: finishing write tid 3 to smithi07929762-17 2024-04-24T01:55:51.891 INFO:tasks.rados.rados.0.smithi079.stdout:171: finishing write tid 4 to smithi07929762-17 2024-04-24T01:55:51.891 INFO:tasks.rados.rados.0.smithi079.stdout:171: finishing write tid 5 to smithi07929762-17 2024-04-24T01:55:51.891 INFO:tasks.rados.rados.0.smithi079.stdout:171: oid 17 updating version 0 to 34 2024-04-24T01:55:51.891 INFO:tasks.rados.rados.0.smithi079.stdout:171: oid 17 version 34 is already newer than 34 2024-04-24T01:55:51.891 INFO:tasks.rados.rados.0.smithi079.stdout:171: oid 17 version 34 is already newer than 34 2024-04-24T01:55:51.891 INFO:tasks.rados.rados.0.smithi079.stdout:171: oid 17 version 34 is already newer than 34 2024-04-24T01:55:51.891 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 17 v 34 (ObjNum 107 snap 9 seq_num 107) dirty exists 2024-04-24T01:55:51.891 INFO:tasks.rados.rados.0.smithi079.stdout:171: left oid 17 (ObjNum 107 snap 9 seq_num 107) 2024-04-24T01:55:51.891 INFO:tasks.rados.rados.0.smithi079.stdout:174: finishing write tid 1 to smithi07929762-39 2024-04-24T01:55:51.891 INFO:tasks.rados.rados.0.smithi079.stdout:176: finishing write tid 1 to smithi07929762-35 2024-04-24T01:55:51.891 INFO:tasks.rados.rados.0.smithi079.stdout:172: finishing write tid 1 to smithi07929762-15 2024-04-24T01:55:51.891 INFO:tasks.rados.rados.0.smithi079.stdout:176: finishing write tid 2 to smithi07929762-35 2024-04-24T01:55:51.892 INFO:tasks.rados.rados.0.smithi079.stdout:174: finishing write tid 2 to smithi07929762-39 2024-04-24T01:55:51.892 INFO:tasks.rados.rados.0.smithi079.stdout:176: finishing write tid 3 to smithi07929762-35 2024-04-24T01:55:51.892 INFO:tasks.rados.rados.0.smithi079.stdout:174: finishing write tid 3 to smithi07929762-39 2024-04-24T01:55:51.892 INFO:tasks.rados.rados.0.smithi079.stdout:176: finishing write tid 4 to smithi07929762-35 2024-04-24T01:55:51.892 INFO:tasks.rados.rados.0.smithi079.stdout:172: finishing write tid 2 to smithi07929762-15 2024-04-24T01:55:51.892 INFO:tasks.rados.rados.0.smithi079.stdout:176: finishing write tid 5 to smithi07929762-35 2024-04-24T01:55:51.892 INFO:tasks.rados.rados.0.smithi079.stdout:176: oid 35 updating version 0 to 21 2024-04-24T01:55:51.892 INFO:tasks.rados.rados.0.smithi079.stdout:176: oid 35 version 21 is already newer than 21 2024-04-24T01:55:51.892 INFO:tasks.rados.rados.0.smithi079.stdout:176: oid 35 version 21 is already newer than 21 2024-04-24T01:55:51.892 INFO:tasks.rados.rados.0.smithi079.stdout:176: oid 35 version 21 is already newer than 21 2024-04-24T01:55:51.892 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 35 v 21 (ObjNum 110 snap 9 seq_num 110) dirty exists 2024-04-24T01:55:51.892 INFO:tasks.rados.rados.0.smithi079.stdout:176: left oid 35 (ObjNum 110 snap 9 seq_num 110) 2024-04-24T01:55:51.892 INFO:tasks.rados.rados.0.smithi079.stdout:174: finishing write tid 4 to smithi07929762-39 2024-04-24T01:55:51.892 INFO:tasks.rados.rados.0.smithi079.stdout:174: finishing write tid 5 to smithi07929762-39 2024-04-24T01:55:51.892 INFO:tasks.rados.rados.0.smithi079.stdout:174: oid 39 updating version 0 to 41 2024-04-24T01:55:51.893 INFO:tasks.rados.rados.0.smithi079.stdout:174: oid 39 version 41 is already newer than 41 2024-04-24T01:55:51.893 INFO:tasks.rados.rados.0.smithi079.stdout:174: oid 39 version 41 is already newer than 38 2024-04-24T01:55:51.893 INFO:tasks.rados.rados.0.smithi079.stdout:174: oid 39 version 41 is already newer than 41 2024-04-24T01:55:51.893 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 39 v 41 (ObjNum 109 snap 9 seq_num 109) dirty exists 2024-04-24T01:55:51.893 INFO:tasks.rados.rados.0.smithi079.stdout:174: left oid 39 (ObjNum 109 snap 9 seq_num 109) 2024-04-24T01:55:51.893 INFO:tasks.rados.rados.0.smithi079.stdout:175: expect (ObjNum 70 snap 3 seq_num 70) 2024-04-24T01:55:52.095 INFO:tasks.rados.rados.0.smithi079.stdout:172: finishing write tid 3 to smithi07929762-15 2024-04-24T01:55:52.095 INFO:tasks.rados.rados.0.smithi079.stdout:172: finishing write tid 4 to smithi07929762-15 2024-04-24T01:55:52.095 INFO:tasks.rados.rados.0.smithi079.stdout:172: finishing write tid 5 to smithi07929762-15 2024-04-24T01:55:52.095 INFO:tasks.rados.rados.0.smithi079.stdout:172: oid 15 updating version 0 to 38 2024-04-24T01:55:52.095 INFO:tasks.rados.rados.0.smithi079.stdout:172: oid 15 version 38 is already newer than 38 2024-04-24T01:55:52.095 INFO:tasks.rados.rados.0.smithi079.stdout:172: oid 15 version 38 is already newer than 38 2024-04-24T01:55:52.095 INFO:tasks.rados.rados.0.smithi079.stdout:172: oid 15 version 38 is already newer than 35 2024-04-24T01:55:52.096 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 15 v 38 (ObjNum 108 snap 9 seq_num 108) dirty exists 2024-04-24T01:55:52.096 INFO:tasks.rados.rados.0.smithi079.stdout:172: left oid 15 (ObjNum 108 snap 9 seq_num 108) 2024-04-24T01:55:52.294 INFO:tasks.rados.rados.0.smithi079.stdout:168: done (9 left) 2024-04-24T01:55:52.295 INFO:tasks.rados.rados.0.smithi079.stdout:169: done (8 left) 2024-04-24T01:55:52.295 INFO:tasks.rados.rados.0.smithi079.stdout:170: done (7 left) 2024-04-24T01:55:52.295 INFO:tasks.rados.rados.0.smithi079.stdout:171: done (6 left) 2024-04-24T01:55:52.295 INFO:tasks.rados.rados.0.smithi079.stdout:172: done (5 left) 2024-04-24T01:55:52.295 INFO:tasks.rados.rados.0.smithi079.stdout:173: done (4 left) 2024-04-24T01:55:52.295 INFO:tasks.rados.rados.0.smithi079.stdout:174: done (3 left) 2024-04-24T01:55:52.295 INFO:tasks.rados.rados.0.smithi079.stdout:175: done (2 left) 2024-04-24T01:55:52.295 INFO:tasks.rados.rados.0.smithi079.stdout:176: done (1 left) 2024-04-24T01:55:52.295 INFO:tasks.rados.rados.0.smithi079.stdout:177: done (0 left) 2024-04-24T01:55:52.295 INFO:tasks.rados.rados.0.smithi079.stdout:178: delete oid 23 current snap is 10 2024-04-24T01:55:52.326 INFO:tasks.rados.rados.0.smithi079.stdout:178: done (0 left) 2024-04-24T01:55:52.326 INFO:tasks.rados.rados.0.smithi079.stdout:179: delete oid 36 current snap is 10 2024-04-24T01:55:52.355 INFO:tasks.rados.rados.0.smithi079.stdout:179: done (0 left) 2024-04-24T01:55:52.356 INFO:tasks.rados.rados.0.smithi079.stdout:180: write oid 41 current snap is 10 2024-04-24T01:55:52.356 INFO:tasks.rados.rados.0.smithi079.stdout:180: seq_num 113 ranges {699191=532079,1779524=567357,2837151=666208} 2024-04-24T01:55:52.362 INFO:tasks.rados.rados.0.smithi079.stdout:180: writing smithi07929762-41 from 699191 to 1231270 tid 1 2024-04-24T01:55:52.368 INFO:tasks.rados.rados.0.smithi079.stdout:180: writing smithi07929762-41 from 1779524 to 2346881 tid 2 2024-04-24T01:55:52.375 INFO:tasks.rados.rados.0.smithi079.stdout:180: writing smithi07929762-41 from 2837151 to 3503359 tid 3 2024-04-24T01:55:52.375 INFO:tasks.rados.rados.0.smithi079.stdout:181: write oid 20 current snap is 10 2024-04-24T01:55:52.375 INFO:tasks.rados.rados.0.smithi079.stdout:181: seq_num 114 ranges {715998=607328,1838006=427368,2864574=477480,3863835=52163} 2024-04-24T01:55:52.382 INFO:tasks.rados.rados.0.smithi079.stdout:181: writing smithi07929762-20 from 715998 to 1323326 tid 1 2024-04-24T01:55:52.387 INFO:tasks.rados.rados.0.smithi079.stdout:181: writing smithi07929762-20 from 1838006 to 2265374 tid 2 2024-04-24T01:55:52.394 INFO:tasks.rados.rados.0.smithi079.stdout:181: writing smithi07929762-20 from 2864574 to 3342054 tid 3 2024-04-24T01:55:52.397 INFO:tasks.rados.rados.0.smithi079.stdout:181: writing smithi07929762-20 from 3863835 to 3915998 tid 4 2024-04-24T01:55:52.397 INFO:tasks.rados.rados.0.smithi079.stdout:182: snap_remove snap 9 2024-04-24T01:55:53.005 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T01:55:53.111 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:53.109+0000 7fbc83a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:53.115 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:53.115+0000 7fbc83a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:53.297 INFO:tasks.rados.rados.0.smithi079.stdout:182: done (2 left) 2024-04-24T01:55:53.297 INFO:tasks.rados.rados.0.smithi079.stdout:183: delete oid 30 current snap is 10 2024-04-24T01:55:53.297 INFO:tasks.rados.rados.0.smithi079.stdout:180: finishing write tid 1 to smithi07929762-41 2024-04-24T01:55:53.297 INFO:tasks.rados.rados.0.smithi079.stdout:180: finishing write tid 2 to smithi07929762-41 2024-04-24T01:55:53.297 INFO:tasks.rados.rados.0.smithi079.stdout:180: finishing write tid 3 to smithi07929762-41 2024-04-24T01:55:53.297 INFO:tasks.rados.rados.0.smithi079.stdout:180: finishing write tid 4 to smithi07929762-41 2024-04-24T01:55:53.297 INFO:tasks.rados.rados.0.smithi079.stdout:180: finishing write tid 5 to smithi07929762-41 2024-04-24T01:55:53.298 INFO:tasks.rados.rados.0.smithi079.stdout:180: oid 41 updating version 0 to 20 2024-04-24T01:55:53.298 INFO:tasks.rados.rados.0.smithi079.stdout:180: oid 41 version 20 is already newer than 20 2024-04-24T01:55:53.298 INFO:tasks.rados.rados.0.smithi079.stdout:180: oid 41 version 20 is already newer than 20 2024-04-24T01:55:53.298 INFO:tasks.rados.rados.0.smithi079.stdout:180: oid 41 version 20 is already newer than 20 2024-04-24T01:55:53.298 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 41 v 20 (ObjNum 113 snap 10 seq_num 113) dirty exists 2024-04-24T01:55:53.298 INFO:tasks.rados.rados.0.smithi079.stdout:180: left oid 41 (ObjNum 113 snap 10 seq_num 113) 2024-04-24T01:55:53.298 INFO:tasks.rados.rados.0.smithi079.stdout:181: finishing write tid 1 to smithi07929762-20 2024-04-24T01:55:53.298 INFO:tasks.rados.rados.0.smithi079.stdout:181: finishing write tid 2 to smithi07929762-20 2024-04-24T01:55:53.298 INFO:tasks.rados.rados.0.smithi079.stdout:181: finishing write tid 3 to smithi07929762-20 2024-04-24T01:55:53.298 INFO:tasks.rados.rados.0.smithi079.stdout:181: finishing write tid 4 to smithi07929762-20 2024-04-24T01:55:53.298 INFO:tasks.rados.rados.0.smithi079.stdout:181: finishing write tid 5 to smithi07929762-20 2024-04-24T01:55:53.298 INFO:tasks.rados.rados.0.smithi079.stdout:181: finishing write tid 6 to smithi07929762-20 2024-04-24T01:55:53.298 INFO:tasks.rados.rados.0.smithi079.stdout:181: oid 20 updating version 0 to 30 2024-04-24T01:55:53.298 INFO:tasks.rados.rados.0.smithi079.stdout:181: oid 20 version 30 is already newer than 30 2024-04-24T01:55:53.298 INFO:tasks.rados.rados.0.smithi079.stdout:181: oid 20 version 30 is already newer than 30 2024-04-24T01:55:53.299 INFO:tasks.rados.rados.0.smithi079.stdout:181: oid 20 version 30 is already newer than 30 2024-04-24T01:55:53.299 INFO:tasks.rados.rados.0.smithi079.stdout:181: oid 20 version 30 is already newer than 30 2024-04-24T01:55:53.299 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 20 v 30 (ObjNum 114 snap 10 seq_num 114) dirty exists 2024-04-24T01:55:53.299 INFO:tasks.rados.rados.0.smithi079.stdout:181: left oid 20 (ObjNum 114 snap 10 seq_num 114) 2024-04-24T01:55:53.364 INFO:tasks.rados.rados.0.smithi079.stdout:180: done (2 left) 2024-04-24T01:55:53.364 INFO:tasks.rados.rados.0.smithi079.stdout:181: done (1 left) 2024-04-24T01:55:53.364 INFO:tasks.rados.rados.0.smithi079.stdout:183: done (0 left) 2024-04-24T01:55:53.364 INFO:tasks.rados.rados.0.smithi079.stdout:184: write (excl) oid 45 current snap is 10 2024-04-24T01:55:53.364 INFO:tasks.rados.rados.0.smithi079.stdout:184: seq_num 116 ranges {749612=474179,1954970=431037,2919420=616377} 2024-04-24T01:55:53.370 INFO:tasks.rados.rados.0.smithi079.stdout:184: writing smithi07929762-45 from 749612 to 1223791 tid 1 2024-04-24T01:55:53.377 INFO:tasks.rados.rados.0.smithi079.stdout:184: writing smithi07929762-45 from 1954970 to 2386007 tid 2 2024-04-24T01:55:53.383 INFO:tasks.rados.rados.0.smithi079.stdout:184: writing smithi07929762-45 from 2919420 to 3535797 tid 3 2024-04-24T01:55:53.383 INFO:tasks.rados.rados.0.smithi079.stdout:185: write (excl) oid 4 current snap is 10 2024-04-24T01:55:53.383 INFO:tasks.rados.rados.0.smithi079.stdout:185: seq_num 117 ranges {766419=549428,2013452=574695,3346843=427649} 2024-04-24T01:55:53.390 INFO:tasks.rados.rados.0.smithi079.stdout:185: writing smithi07929762-4 from 766419 to 1315847 tid 1 2024-04-24T01:55:53.399 INFO:tasks.rados.rados.0.smithi079.stdout:185: writing smithi07929762-4 from 2013452 to 2588147 tid 2 2024-04-24T01:55:53.409 INFO:tasks.rados.rados.0.smithi079.stdout:185: writing smithi07929762-4 from 3346843 to 3774492 tid 3 2024-04-24T01:55:53.409 INFO:tasks.rados.rados.0.smithi079.stdout:186: delete oid 30 current snap is 10 2024-04-24T01:55:53.416 INFO:tasks.rados.rados.0.smithi079.stdout:186: done (2 left) 2024-04-24T01:55:53.416 INFO:tasks.rados.rados.0.smithi079.stdout:187: write (excl) oid 3 current snap is 10 2024-04-24T01:55:53.416 INFO:tasks.rados.rados.0.smithi079.stdout:187: seq_num 119 ranges {400033=699926,1614063=385970} 2024-04-24T01:55:53.423 INFO:tasks.rados.rados.0.smithi079.stdout:187: writing smithi07929762-3 from 400033 to 1099959 tid 1 2024-04-24T01:55:53.428 INFO:tasks.rados.rados.0.smithi079.stdout:187: writing smithi07929762-3 from 1614063 to 2000033 tid 2 2024-04-24T01:55:53.429 INFO:tasks.rados.rados.0.smithi079.stdout:188: write oid 26 current snap is 10 2024-04-24T01:55:53.429 INFO:tasks.rados.rados.0.smithi079.stdout:188: seq_num 120 ranges {416840=775175,1672545=344295} 2024-04-24T01:55:53.435 INFO:tasks.rados.rados.0.smithi079.stdout:188: writing smithi07929762-26 from 416840 to 1192015 tid 1 2024-04-24T01:55:53.440 INFO:tasks.rados.rados.0.smithi079.stdout:188: writing smithi07929762-26 from 1672545 to 2016840 tid 2 2024-04-24T01:55:53.440 INFO:tasks.rados.rados.0.smithi079.stdout:189: snap_create 2024-04-24T01:55:53.440 INFO:tasks.rados.rados.0.smithi079.stdout:184: finishing write tid 1 to smithi07929762-45 2024-04-24T01:55:53.471 INFO:tasks.rados.rados.0.smithi079.stdout:184: finishing write tid 2 to smithi07929762-45 2024-04-24T01:55:53.501 INFO:tasks.rados.rados.0.smithi079.stdout:184: finishing write tid 3 to smithi07929762-45 2024-04-24T01:55:53.503 INFO:tasks.rados.rados.0.smithi079.stdout:188: finishing write tid 1 to smithi07929762-26 2024-04-24T01:55:53.511 INFO:tasks.rados.rados.0.smithi079.stdout:188: finishing write tid 2 to smithi07929762-26 2024-04-24T01:55:53.514 INFO:tasks.rados.rados.0.smithi079.stdout:184: finishing write tid 4 to smithi07929762-45 2024-04-24T01:55:53.515 INFO:tasks.rados.rados.0.smithi079.stdout:184: finishing write tid 5 to smithi07929762-45 2024-04-24T01:55:53.515 INFO:tasks.rados.rados.0.smithi079.stdout:184: oid 45 updating version 0 to 44 2024-04-24T01:55:53.516 INFO:tasks.rados.rados.0.smithi079.stdout:184: oid 45 version 44 is already newer than 44 2024-04-24T01:55:53.516 INFO:tasks.rados.rados.0.smithi079.stdout:184: oid 45 version 44 is already newer than 44 2024-04-24T01:55:53.516 INFO:tasks.rados.rados.0.smithi079.stdout:184: oid 45 version 44 is already newer than 41 2024-04-24T01:55:53.516 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 45 v 44 (ObjNum 116 snap 10 seq_num 116) dirty exists 2024-04-24T01:55:53.516 INFO:tasks.rados.rados.0.smithi079.stdout:184: left oid 45 (ObjNum 116 snap 10 seq_num 116) 2024-04-24T01:55:53.518 INFO:tasks.rados.rados.0.smithi079.stdout:188: finishing write tid 3 to smithi07929762-26 2024-04-24T01:55:53.520 INFO:tasks.rados.rados.0.smithi079.stdout:187: finishing write tid 1 to smithi07929762-3 2024-04-24T01:55:53.521 INFO:tasks.rados.rados.0.smithi079.stdout:188: finishing write tid 4 to smithi07929762-26 2024-04-24T01:55:53.521 INFO:tasks.rados.rados.0.smithi079.stdout:188: oid 26 updating version 0 to 42 2024-04-24T01:55:53.521 INFO:tasks.rados.rados.0.smithi079.stdout:188: oid 26 version 42 is already newer than 42 2024-04-24T01:55:53.521 INFO:tasks.rados.rados.0.smithi079.stdout:188: oid 26 version 42 is already newer than 42 2024-04-24T01:55:53.521 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 26 v 42 (ObjNum 120 snap 10 seq_num 120) dirty exists 2024-04-24T01:55:53.521 INFO:tasks.rados.rados.0.smithi079.stdout:188: left oid 26 (ObjNum 120 snap 10 seq_num 120) 2024-04-24T01:55:53.521 INFO:tasks.rados.rados.0.smithi079.stdout:185: finishing write tid 1 to smithi07929762-4 2024-04-24T01:55:53.549 INFO:tasks.rados.rados.0.smithi079.stdout:187: finishing write tid 2 to smithi07929762-3 2024-04-24T01:55:53.558 INFO:tasks.rados.rados.0.smithi079.stdout:185: finishing write tid 2 to smithi07929762-4 2024-04-24T01:55:53.558 INFO:tasks.rados.rados.0.smithi079.stdout:187: finishing write tid 3 to smithi07929762-3 2024-04-24T01:55:53.559 INFO:tasks.rados.rados.0.smithi079.stdout:187: finishing write tid 4 to smithi07929762-3 2024-04-24T01:55:53.559 INFO:tasks.rados.rados.0.smithi079.stdout:187: oid 3 updating version 0 to 53 2024-04-24T01:55:53.559 INFO:tasks.rados.rados.0.smithi079.stdout:187: oid 3 version 53 is already newer than 53 2024-04-24T01:55:53.559 INFO:tasks.rados.rados.0.smithi079.stdout:187: oid 3 version 53 is already newer than 51 2024-04-24T01:55:53.560 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 3 v 53 (ObjNum 119 snap 10 seq_num 119) dirty exists 2024-04-24T01:55:53.560 INFO:tasks.rados.rados.0.smithi079.stdout:187: left oid 3 (ObjNum 119 snap 10 seq_num 119) 2024-04-24T01:55:53.564 INFO:tasks.rados.rados.0.smithi079.stdout:185: finishing write tid 3 to smithi07929762-4 2024-04-24T01:55:53.567 INFO:tasks.rados.rados.0.smithi079.stdout:185: finishing write tid 4 to smithi07929762-4 2024-04-24T01:55:53.568 INFO:tasks.rados.rados.0.smithi079.stdout:185: finishing write tid 5 to smithi07929762-4 2024-04-24T01:55:53.568 INFO:tasks.rados.rados.0.smithi079.stdout:185: oid 4 updating version 0 to 9 2024-04-24T01:55:53.568 INFO:tasks.rados.rados.0.smithi079.stdout:185: oid 4 updating version 9 to 12 2024-04-24T01:55:53.568 INFO:tasks.rados.rados.0.smithi079.stdout:185: oid 4 version 12 is already newer than 12 2024-04-24T01:55:53.568 INFO:tasks.rados.rados.0.smithi079.stdout:185: oid 4 version 12 is already newer than 12 2024-04-24T01:55:53.568 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 4 v 12 (ObjNum 117 snap 10 seq_num 117) dirty exists 2024-04-24T01:55:53.568 INFO:tasks.rados.rados.0.smithi079.stdout:185: left oid 4 (ObjNum 117 snap 10 seq_num 117) 2024-04-24T01:55:54.298 INFO:tasks.rados.rados.0.smithi079.stdout:184: done (4 left) 2024-04-24T01:55:54.299 INFO:tasks.rados.rados.0.smithi079.stdout:185: done (3 left) 2024-04-24T01:55:54.299 INFO:tasks.rados.rados.0.smithi079.stdout:187: done (2 left) 2024-04-24T01:55:54.299 INFO:tasks.rados.rados.0.smithi079.stdout:188: done (1 left) 2024-04-24T01:55:54.299 INFO:tasks.rados.rados.0.smithi079.stdout:189: done (0 left) 2024-04-24T01:55:54.299 INFO:tasks.rados.rados.0.smithi079.stdout:190: snap_create 2024-04-24T01:55:55.093 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-04-24T01:55:55.093 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-04-24T01:55:55.094 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-04-24T01:55:55.094 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-04-24T01:55:55.205 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:55.199+0000 7f4f0fcad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:55.209 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:55.208+0000 7f4f0fcad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:55.212 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T01:55:55.225 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T01:55:55.225 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T01:55:55.225 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T01:55:55.225 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T01:55:55.225 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T01:55:55.225 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T01:55:55.225 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T01:55:55.225 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T01:55:55.225 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T01:55:55.225 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T01:55:55.225 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T01:55:55.225 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T01:55:55.228 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T01:55:55.228 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T01:55:55.228 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-04-24T01:55:55.304 INFO:tasks.rados.rados.0.smithi079.stdout:190: done (0 left) 2024-04-24T01:55:55.304 INFO:tasks.rados.rados.0.smithi079.stdout:191: read oid 2 snap -1 2024-04-24T01:55:55.304 INFO:tasks.rados.rados.0.smithi079.stdout:191: expect (ObjNum 106 snap 9 seq_num 106) 2024-04-24T01:55:55.304 INFO:tasks.rados.rados.0.smithi079.stdout:192: snap_create 2024-04-24T01:55:55.320 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:55.318+0000 7f34dd75d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:55.324 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:55.323+0000 7f34dd75d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:55.364 INFO:tasks.rados.rados.0.smithi079.stdout:191: expect (ObjNum 106 snap 9 seq_num 106) 2024-04-24T01:55:55.391 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:55.385+0000 7fc2c926d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:55.394 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:55.394+0000 7fc2c926d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:55.411 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T01:55:55.411 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T01:55:55.411 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T01:55:55.411 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T01:55:55.412 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T01:55:55.412 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T01:55:55.412 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T01:55:55.412 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T01:55:55.412 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T01:55:55.412 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T01:55:55.412 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T01:55:55.412 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T01:55:55.414 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T01:55:55.414 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T01:55:55.415 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-04-24T01:55:55.580 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:55.574+0000 7f173caad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:55.584 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:55.583+0000 7f173caad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:55.600 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T01:55:55.601 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T01:55:55.601 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T01:55:55.601 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T01:55:55.601 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T01:55:55.601 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T01:55:55.601 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T01:55:55.601 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T01:55:55.601 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T01:55:55.601 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T01:55:55.601 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T01:55:55.601 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T01:55:55.603 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T01:55:55.604 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T01:55:55.604 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-04-24T01:55:55.711 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:55.705+0000 7fece36ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:55.715 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:55.714+0000 7fece36ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:55.731 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T01:55:55.731 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T01:55:55.731 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T01:55:55.731 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T01:55:55.731 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T01:55:55.731 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T01:55:55.731 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T01:55:55.732 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T01:55:55.732 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T01:55:55.732 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T01:55:55.732 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T01:55:55.732 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T01:55:55.734 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T01:55:55.734 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T01:55:56.304 INFO:tasks.rados.rados.0.smithi079.stdout:191: done (1 left) 2024-04-24T01:55:56.304 INFO:tasks.rados.rados.0.smithi079.stdout:192: done (0 left) 2024-04-24T01:55:56.304 INFO:tasks.rados.rados.0.smithi079.stdout:193: snap_create 2024-04-24T01:55:57.311 INFO:tasks.rados.rados.0.smithi079.stdout:193: done (0 left) 2024-04-24T01:55:57.311 INFO:tasks.rados.rados.0.smithi079.stdout:194: delete oid 46 current snap is 14 2024-04-24T01:55:57.405 INFO:tasks.rados.rados.0.smithi079.stdout:194: done (0 left) 2024-04-24T01:55:57.406 INFO:tasks.rados.rados.0.smithi079.stdout:195: snap_remove snap 13 2024-04-24T01:55:57.433 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T01:55:57.538 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:57.536+0000 7f30538ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:57.542 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:57.541+0000 7f30538ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:58.311 INFO:tasks.rados.rados.0.smithi079.stdout:195: done (0 left) 2024-04-24T01:55:58.311 INFO:tasks.rados.rados.0.smithi079.stdout:196: write (excl) oid 40 current snap is 14 2024-04-24T01:55:58.311 INFO:tasks.rados.rados.0.smithi079.stdout:196: seq_num 122 ranges {450454=642026,1505862=544592} 2024-04-24T01:55:58.317 INFO:tasks.rados.rados.0.smithi079.stdout:196: writing smithi07929762-40 from 450454 to 1092480 tid 1 2024-04-24T01:55:58.323 INFO:tasks.rados.rados.0.smithi079.stdout:196: writing smithi07929762-40 from 1505862 to 2050454 tid 2 2024-04-24T01:55:58.323 INFO:tasks.rados.rados.0.smithi079.stdout:197: read oid 8 snap 12 2024-04-24T01:55:58.323 INFO:tasks.rados.rados.0.smithi079.stdout:197: expect deleted 2024-04-24T01:55:58.323 INFO:tasks.rados.rados.0.smithi079.stdout:198: delete oid 17 current snap is 14 2024-04-24T01:55:58.412 INFO:tasks.rados.rados.0.smithi079.stdout:196: finishing write tid 1 to smithi07929762-40 2024-04-24T01:55:58.417 INFO:tasks.rados.rados.0.smithi079.stdout:197: done (2 left) 2024-04-24T01:55:58.417 INFO:tasks.rados.rados.0.smithi079.stdout:198: done (1 left) 2024-04-24T01:55:58.417 INFO:tasks.rados.rados.0.smithi079.stdout:199: snap_create 2024-04-24T01:55:58.430 INFO:tasks.rados.rados.0.smithi079.stdout:196: finishing write tid 2 to smithi07929762-40 2024-04-24T01:55:58.437 INFO:tasks.rados.rados.0.smithi079.stdout:196: finishing write tid 3 to smithi07929762-40 2024-04-24T01:55:58.438 INFO:tasks.rados.rados.0.smithi079.stdout:196: finishing write tid 4 to smithi07929762-40 2024-04-24T01:55:58.438 INFO:tasks.rados.rados.0.smithi079.stdout:196: oid 40 updating version 0 to 36 2024-04-24T01:55:58.438 INFO:tasks.rados.rados.0.smithi079.stdout:196: oid 40 version 36 is already newer than 36 2024-04-24T01:55:58.438 INFO:tasks.rados.rados.0.smithi079.stdout:196: oid 40 version 36 is already newer than 34 2024-04-24T01:55:58.439 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 40 v 36 (ObjNum 122 snap 14 seq_num 122) dirty exists 2024-04-24T01:55:58.439 INFO:tasks.rados.rados.0.smithi079.stdout:196: left oid 40 (ObjNum 122 snap 14 seq_num 122) 2024-04-24T01:55:59.318 INFO:tasks.rados.rados.0.smithi079.stdout:196: done (1 left) 2024-04-24T01:55:59.318 INFO:tasks.rados.rados.0.smithi079.stdout:199: done (0 left) 2024-04-24T01:55:59.318 INFO:tasks.rados.rados.0.smithi079.stdout:200: write (excl) oid 33 current snap is 15 2024-04-24T01:55:59.318 INFO:tasks.rados.rados.0.smithi079.stdout:200: seq_num 124 ranges {484068=792524,1906473=177595} 2024-04-24T01:55:59.325 INFO:tasks.rados.rados.0.smithi079.stdout:200: writing smithi07929762-33 from 484068 to 1276592 tid 1 2024-04-24T01:55:59.330 INFO:tasks.rados.rados.0.smithi079.stdout:200: writing smithi07929762-33 from 1906473 to 2084068 tid 2 2024-04-24T01:55:59.330 INFO:tasks.rados.rados.0.smithi079.stdout:201: read oid 46 snap 12 2024-04-24T01:55:59.330 INFO:tasks.rados.rados.0.smithi079.stdout:201: expect (ObjNum 62 snap 2 seq_num 62) 2024-04-24T01:55:59.330 INFO:tasks.rados.rados.0.smithi079.stdout:202: delete oid 19 current snap is 15 2024-04-24T01:55:59.408 INFO:tasks.rados.rados.0.smithi079.stdout:202: done (2 left) 2024-04-24T01:55:59.408 INFO:tasks.rados.rados.0.smithi079.stdout:203: snap_remove snap 14 2024-04-24T01:55:59.638 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T01:55:59.743 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:59.741+0000 7efefc206640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:55:59.747 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:55:59.746+0000 7efefc206640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:00.319 INFO:tasks.rados.rados.0.smithi079.stdout:203: done (2 left) 2024-04-24T01:56:00.319 INFO:tasks.rados.rados.0.smithi079.stdout:204: read oid 32 snap -1 2024-04-24T01:56:00.319 INFO:tasks.rados.rados.0.smithi079.stdout:204: expect (ObjNum 31 snap 0 seq_num 31) 2024-04-24T01:56:00.319 INFO:tasks.rados.rados.0.smithi079.stdout:205: write (excl) oid 5 current snap is 15 2024-04-24T01:56:00.319 INFO:tasks.rados.rados.0.smithi079.stdout:200: finishing write tid 1 to smithi07929762-33 2024-04-24T01:56:00.319 INFO:tasks.rados.rados.0.smithi079.stdout:205: seq_num 126 ranges {517682=543022,1623437=494245} 2024-04-24T01:56:00.324 INFO:tasks.rados.rados.0.smithi079.stdout:205: writing smithi07929762-5 from 517682 to 1060704 tid 1 2024-04-24T01:56:00.330 INFO:tasks.rados.rados.0.smithi079.stdout:205: writing smithi07929762-5 from 1623437 to 2117682 tid 2 2024-04-24T01:56:00.330 INFO:tasks.rados.rados.0.smithi079.stdout:206: read oid 31 snap -1 2024-04-24T01:56:00.330 INFO:tasks.rados.rados.0.smithi079.stdout:206: expect (ObjNum 30 snap 0 seq_num 30) 2024-04-24T01:56:00.330 INFO:tasks.rados.rados.0.smithi079.stdout:207: snap_remove snap 10 2024-04-24T01:56:00.735 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-04-24T01:56:00.735 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-04-24T01:56:00.736 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T01:56:00.842 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:00.837+0000 7f9444eed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:00.846 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:00.845+0000 7f9444eed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:01.033 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:56:01.034 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-24T01:56:01.046 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":61,"stamp":"2024-04-24T01:56:00.943482+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":121952494,"num_objects":45,"num_object_clones":2,"num_object_copies":135,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":558,"num_read_kb":254077,"num_write":603,"num_write_kb":131273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":556,"ondisk_log_size":556,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":315752,"kb_used_data":206784,"kb_used_omap":0,"kb_used_meta":107264,"kb_avail":377171608,"statfs":{"total":386547056640,"available":386223726592,"internally_reserved":0,"allocated":211746816,"data_stored":210759321,"data_compressed":89889,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109838336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-8521050,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":8,"num_read_kb":10294,"num_write":6,"num_write_kb":1095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001084"},"pg_stats":[{"pgid":"3.f","version":"19'6","reported_seq":106,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-24T01:55:57.348355+0000","last_change":"2024-04-24T01:55:53.449421+0000","last_active":"2024-04-24T01:55:57.348355+0000","last_peered":"2024-04-24T01:55:57.348355+0000","last_clean":"2024-04-24T01:55:57.348355+0000","last_became_active":"2024-04-24T01:55:30.386726+0000","last_became_peered":"2024-04-24T01:55:30.386726+0000","last_unstale":"2024-04-24T01:55:57.348355+0000","last_undegraded":"2024-04-24T01:55:57.348355+0000","last_fullsized":"2024-04-24T01:55:57.348355+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5310946,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":22,"num_read_kb":17170,"num_write":8,"num_write_kb":2595,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"39'44","reported_seq":148,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T01:55:58.337827+0000","last_change":"2024-04-24T01:55:58.352149+0000","last_active":"2024-04-24T01:55:58.337827+0000","last_peered":"2024-04-24T01:55:58.337827+0000","last_clean":"2024-04-24T01:55:58.337827+0000","last_became_active":"2024-04-24T01:55:30.321866+0000","last_became_peered":"2024-04-24T01:55:30.321866+0000","last_unstale":"2024-04-24T01:55:58.337827+0000","last_undegraded":"2024-04-24T01:55:58.337827+0000","last_fullsized":"2024-04-24T01:55:58.337827+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":3,"snaptrim_duration":0,"stat_sum":{"num_bytes":9596698,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":38,"num_read_kb":19722,"num_write":40,"num_write_kb":11360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"37'42","reported_seq":133,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-24T01:55:57.352693+0000","last_change":"2024-04-24T01:55:53.312966+0000","last_active":"2024-04-24T01:55:57.352693+0000","last_peered":"2024-04-24T01:55:57.352693+0000","last_clean":"2024-04-24T01:55:57.352693+0000","last_became_active":"2024-04-24T01:55:30.320186+0000","last_became_peered":"2024-04-24T01:55:30.320186+0000","last_unstale":"2024-04-24T01:55:57.352693+0000","last_undegraded":"2024-04-24T01:55:57.352693+0000","last_fullsized":"2024-04-24T01:55:57.352693+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0,"stat_sum":{"num_bytes":10554697,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":9,"num_write":43,"num_write_kb":10987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"37'45","reported_seq":145,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-24T01:55:57.353957+0000","last_change":"2024-04-24T01:55:53.449125+0000","last_active":"2024-04-24T01:55:57.353957+0000","last_peered":"2024-04-24T01:55:57.353957+0000","last_clean":"2024-04-24T01:55:57.353957+0000","last_became_active":"2024-04-24T01:55:30.288081+0000","last_became_peered":"2024-04-24T01:55:30.288081+0000","last_unstale":"2024-04-24T01:55:57.353957+0000","last_undegraded":"2024-04-24T01:55:57.353957+0000","last_fullsized":"2024-04-24T01:55:57.353957+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0,"stat_sum":{"num_bytes":5882384,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":44,"num_read_kb":31023,"num_write":33,"num_write_kb":9612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"39'23","reported_seq":127,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T01:55:58.333222+0000","last_change":"2024-04-24T01:55:58.342302+0000","last_active":"2024-04-24T01:55:58.333222+0000","last_peered":"2024-04-24T01:55:58.333222+0000","last_clean":"2024-04-24T01:55:58.333222+0000","last_became_active":"2024-04-24T01:55:30.333815+0000","last_became_peered":"2024-04-24T01:55:30.333815+0000","last_unstale":"2024-04-24T01:55:58.333222+0000","last_undegraded":"2024-04-24T01:55:58.333222+0000","last_fullsized":"2024-04-24T01:55:58.333222+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0,"stat_sum":{"num_bytes":3243718,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":22,"num_read_kb":11072,"num_write":25,"num_write_kb":6861,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"39'53","reported_seq":164,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-24T01:55:57.345487+0000","last_change":"2024-04-24T01:55:53.449647+0000","last_active":"2024-04-24T01:55:57.345487+0000","last_peered":"2024-04-24T01:55:57.345487+0000","last_clean":"2024-04-24T01:55:57.345487+0000","last_became_active":"2024-04-24T01:55:30.315075+0000","last_became_peered":"2024-04-24T01:55:30.315075+0000","last_unstale":"2024-04-24T01:55:57.345487+0000","last_undegraded":"2024-04-24T01:55:57.345487+0000","last_fullsized":"2024-04-24T01:55:57.345487+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0,"stat_sum":{"num_bytes":15260591,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":55,"num_read_kb":31525,"num_write":57,"num_write_kb":16334,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"39'12","reported_seq":106,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-24T01:55:57.361921+0000","last_change":"2024-04-24T01:55:53.454209+0000","last_active":"2024-04-24T01:55:57.361921+0000","last_peered":"2024-04-24T01:55:57.361921+0000","last_clean":"2024-04-24T01:55:57.361921+0000","last_became_active":"2024-04-24T01:55:30.350300+0000","last_became_peered":"2024-04-24T01:55:30.350300+0000","last_unstale":"2024-04-24T01:55:57.361921+0000","last_undegraded":"2024-04-24T01:55:57.361921+0000","last_fullsized":"2024-04-24T01:55:57.361921+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3966419,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":5003,"num_write":14,"num_write_kb":3526,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"18'4","reported_seq":92,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-24T01:55:57.373012+0000","last_change":"2024-04-24T01:55:53.322601+0000","last_active":"2024-04-24T01:55:57.373012+0000","last_peered":"2024-04-24T01:55:57.373012+0000","last_clean":"2024-04-24T01:55:57.373012+0000","last_became_active":"2024-04-24T01:55:30.376813+0000","last_became_peered":"2024-04-24T01:55:30.376813+0000","last_unstale":"2024-04-24T01:55:57.373012+0000","last_undegraded":"2024-04-24T01:55:57.373012+0000","last_fullsized":"2024-04-24T01:55:57.373012+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'97","reported_seq":226,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T01:55:58.321255+0000","last_change":"2024-04-24T01:55:16.196470+0000","last_active":"2024-04-24T01:55:58.321255+0000","last_peered":"2024-04-24T01:55:58.321255+0000","last_clean":"2024-04-24T01:55:58.321255+0000","last_became_active":"2024-04-24T01:55:16.190464+0000","last_became_peered":"2024-04-24T01:55:16.190464+0000","last_unstale":"2024-04-24T01:55:58.321255+0000","last_undegraded":"2024-04-24T01:55:58.321255+0000","last_fullsized":"2024-04-24T01:55:58.321255+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:15.127115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:15.127115+0000","last_clean_scrub_stamp":"2024-04-24T01:55:15.127115+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":86,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-24T01:55:57.365203+0000","last_change":"2024-04-24T01:55:18.236148+0000","last_active":"2024-04-24T01:55:57.365203+0000","last_peered":"2024-04-24T01:55:57.365203+0000","last_clean":"2024-04-24T01:55:57.365203+0000","last_became_active":"2024-04-24T01:55:18.230154+0000","last_became_peered":"2024-04-24T01:55:18.230154+0000","last_unstale":"2024-04-24T01:55:57.365203+0000","last_undegraded":"2024-04-24T01:55:57.365203+0000","last_fullsized":"2024-04-24T01:55:57.365203+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"39'42","reported_seq":150,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-24T01:55:57.370739+0000","last_change":"2024-04-24T01:55:53.323557+0000","last_active":"2024-04-24T01:55:57.370739+0000","last_peered":"2024-04-24T01:55:57.370739+0000","last_clean":"2024-04-24T01:55:57.370739+0000","last_became_active":"2024-04-24T01:55:30.316222+0000","last_became_peered":"2024-04-24T01:55:30.316222+0000","last_unstale":"2024-04-24T01:55:57.370739+0000","last_undegraded":"2024-04-24T01:55:57.370739+0000","last_fullsized":"2024-04-24T01:55:57.370739+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0,"stat_sum":{"num_bytes":11260591,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":47,"num_read_kb":25610,"num_write":46,"num_write_kb":13556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":88,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-24T01:55:57.363009+0000","last_change":"2024-04-24T01:55:18.265905+0000","last_active":"2024-04-24T01:55:57.363009+0000","last_peered":"2024-04-24T01:55:57.363009+0000","last_clean":"2024-04-24T01:55:57.363009+0000","last_became_active":"2024-04-24T01:55:18.257020+0000","last_became_peered":"2024-04-24T01:55:18.257020+0000","last_unstale":"2024-04-24T01:55:57.363009+0000","last_undegraded":"2024-04-24T01:55:57.363009+0000","last_fullsized":"2024-04-24T01:55:57.363009+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"37'21","reported_seq":134,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T01:55:58.325013+0000","last_change":"2024-04-24T01:55:58.331324+0000","last_active":"2024-04-24T01:55:58.325013+0000","last_peered":"2024-04-24T01:55:58.325013+0000","last_clean":"2024-04-24T01:55:58.325013+0000","last_became_active":"2024-04-24T01:55:30.344129+0000","last_became_peered":"2024-04-24T01:55:30.344129+0000","last_unstale":"2024-04-24T01:55:58.325013+0000","last_undegraded":"2024-04-24T01:55:58.325013+0000","last_fullsized":"2024-04-24T01:55:58.325013+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9092488,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":19957,"num_write":27,"num_write_kb":7404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-24T01:55:57.350816+0000","last_change":"2024-04-24T01:55:18.215251+0000","last_active":"2024-04-24T01:55:57.350816+0000","last_peered":"2024-04-24T01:55:57.350816+0000","last_clean":"2024-04-24T01:55:57.350816+0000","last_became_active":"2024-04-24T01:55:18.206605+0000","last_became_peered":"2024-04-24T01:55:18.206605+0000","last_unstale":"2024-04-24T01:55:57.350816+0000","last_undegraded":"2024-04-24T01:55:57.350816+0000","last_fullsized":"2024-04-24T01:55:57.350816+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"39'22","reported_seq":122,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T01:55:58.329809+0000","last_change":"2024-04-24T01:55:58.338079+0000","last_active":"2024-04-24T01:55:58.329809+0000","last_peered":"2024-04-24T01:55:58.329809+0000","last_clean":"2024-04-24T01:55:58.329809+0000","last_became_active":"2024-04-24T01:55:30.401366+0000","last_became_peered":"2024-04-24T01:55:30.401366+0000","last_unstale":"2024-04-24T01:55:58.329809+0000","last_undegraded":"2024-04-24T01:55:58.329809+0000","last_fullsized":"2024-04-24T01:55:58.329809+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0,"stat_sum":{"num_bytes":7042067,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":9364,"num_write":18,"num_write_kb":5026,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":88,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-24T01:55:57.371562+0000","last_change":"2024-04-24T01:55:18.230893+0000","last_active":"2024-04-24T01:55:57.371562+0000","last_peered":"2024-04-24T01:55:57.371562+0000","last_clean":"2024-04-24T01:55:57.371562+0000","last_became_active":"2024-04-24T01:55:18.225188+0000","last_became_peered":"2024-04-24T01:55:18.225188+0000","last_unstale":"2024-04-24T01:55:57.371562+0000","last_undegraded":"2024-04-24T01:55:57.371562+0000","last_fullsized":"2024-04-24T01:55:57.371562+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"37'41","reported_seq":138,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T01:55:58.346600+0000","last_change":"2024-04-24T01:55:58.352320+0000","last_active":"2024-04-24T01:55:58.346600+0000","last_peered":"2024-04-24T01:55:58.346600+0000","last_clean":"2024-04-24T01:55:58.346600+0000","last_became_active":"2024-04-24T01:55:30.343318+0000","last_became_peered":"2024-04-24T01:55:30.343318+0000","last_unstale":"2024-04-24T01:55:58.346600+0000","last_undegraded":"2024-04-24T01:55:58.346600+0000","last_fullsized":"2024-04-24T01:55:58.346600+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":3,"snaptrim_duration":0,"stat_sum":{"num_bytes":10722767,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":26,"num_read_kb":11814,"num_write":36,"num_write_kb":11038,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"39'32","reported_seq":140,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-24T01:55:57.369084+0000","last_change":"2024-04-24T01:55:53.367233+0000","last_active":"2024-04-24T01:55:57.369084+0000","last_peered":"2024-04-24T01:55:57.369084+0000","last_clean":"2024-04-24T01:55:57.369084+0000","last_became_active":"2024-04-24T01:55:30.345308+0000","last_became_peered":"2024-04-24T01:55:30.345308+0000","last_unstale":"2024-04-24T01:55:57.369084+0000","last_undegraded":"2024-04-24T01:55:57.369084+0000","last_fullsized":"2024-04-24T01:55:57.369084+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0,"stat_sum":{"num_bytes":12638732,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":48,"num_read_kb":30123,"num_write":33,"num_write_kb":8249,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":89,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T01:55:58.354141+0000","last_change":"2024-04-24T01:55:18.224238+0000","last_active":"2024-04-24T01:55:58.354141+0000","last_peered":"2024-04-24T01:55:58.354141+0000","last_clean":"2024-04-24T01:55:58.354141+0000","last_became_active":"2024-04-24T01:55:18.219131+0000","last_became_peered":"2024-04-24T01:55:18.219131+0000","last_unstale":"2024-04-24T01:55:58.354141+0000","last_undegraded":"2024-04-24T01:55:58.354141+0000","last_fullsized":"2024-04-24T01:55:58.354141+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"37'15","reported_seq":116,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T01:55:58.352985+0000","last_change":"2024-04-24T01:55:58.359950+0000","last_active":"2024-04-24T01:55:58.352985+0000","last_peered":"2024-04-24T01:55:58.352985+0000","last_clean":"2024-04-24T01:55:58.352985+0000","last_became_active":"2024-04-24T01:55:30.319958+0000","last_became_peered":"2024-04-24T01:55:30.319958+0000","last_unstale":"2024-04-24T01:55:58.352985+0000","last_undegraded":"2024-04-24T01:55:58.352985+0000","last_fullsized":"2024-04-24T01:55:58.352985+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3781542,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":19604,"num_write":19,"num_write_kb":5672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":93,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T01:55:58.334302+0000","last_change":"2024-04-24T01:55:18.235763+0000","last_active":"2024-04-24T01:55:58.334302+0000","last_peered":"2024-04-24T01:55:58.334302+0000","last_clean":"2024-04-24T01:55:58.334302+0000","last_became_active":"2024-04-24T01:55:18.229175+0000","last_became_peered":"2024-04-24T01:55:18.229175+0000","last_unstale":"2024-04-24T01:55:58.334302+0000","last_undegraded":"2024-04-24T01:55:58.334302+0000","last_fullsized":"2024-04-24T01:55:58.334302+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"39'19","reported_seq":124,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T01:55:58.325932+0000","last_change":"2024-04-24T01:55:58.332090+0000","last_active":"2024-04-24T01:55:58.325932+0000","last_peered":"2024-04-24T01:55:58.325932+0000","last_clean":"2024-04-24T01:55:58.325932+0000","last_became_active":"2024-04-24T01:55:30.372690+0000","last_became_peered":"2024-04-24T01:55:30.372690+0000","last_unstale":"2024-04-24T01:55:58.325932+0000","last_undegraded":"2024-04-24T01:55:58.325932+0000","last_fullsized":"2024-04-24T01:55:58.325932+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0,"stat_sum":{"num_bytes":3462209,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":26,"num_read_kb":16700,"num_write":19,"num_write_kb":5298,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":89,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T01:55:58.354196+0000","last_change":"2024-04-24T01:55:18.278351+0000","last_active":"2024-04-24T01:55:58.354196+0000","last_peered":"2024-04-24T01:55:58.354196+0000","last_clean":"2024-04-24T01:55:58.354196+0000","last_became_active":"2024-04-24T01:55:18.273668+0000","last_became_peered":"2024-04-24T01:55:18.273668+0000","last_unstale":"2024-04-24T01:55:58.354196+0000","last_undegraded":"2024-04-24T01:55:58.354196+0000","last_fullsized":"2024-04-24T01:55:58.354196+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"37'38","reported_seq":139,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T01:55:58.362691+0000","last_change":"2024-04-24T01:55:58.362691+0000","last_active":"2024-04-24T01:55:58.362691+0000","last_peered":"2024-04-24T01:55:58.362691+0000","last_clean":"2024-04-24T01:55:58.362691+0000","last_became_active":"2024-04-24T01:55:30.363123+0000","last_became_peered":"2024-04-24T01:55:30.363123+0000","last_unstale":"2024-04-24T01:55:58.362691+0000","last_undegraded":"2024-04-24T01:55:58.362691+0000","last_fullsized":"2024-04-24T01:55:58.362691+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0,"stat_sum":{"num_bytes":9546277,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":23,"num_read_kb":5289,"num_write":43,"num_write_kb":11013,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":86,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-24T01:55:57.361803+0000","last_change":"2024-04-24T01:55:18.241117+0000","last_active":"2024-04-24T01:55:57.361803+0000","last_peered":"2024-04-24T01:55:57.361803+0000","last_clean":"2024-04-24T01:55:57.361803+0000","last_became_active":"2024-04-24T01:55:18.234313+0000","last_became_peered":"2024-04-24T01:55:18.234313+0000","last_unstale":"2024-04-24T01:55:57.361803+0000","last_undegraded":"2024-04-24T01:55:57.361803+0000","last_fullsized":"2024-04-24T01:55:57.361803+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":121362126,"num_objects":43,"num_object_clones":2,"num_object_copies":129,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":455,"num_read_kb":253986,"num_write":466,"num_write_kb":129379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":212578304,"data_stored":211405561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":459,"ondisk_log_size":459,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":1}],"osd_stats":[{"osd":3,"up_from":10,"seq":42949673004,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":86744,"kb_used_data":58248,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94285096,"statfs":{"total":96636764160,"available":96547938304,"internally_reserved":0,"allocated":59645952,"data_stored":59471101,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673004,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":74448,"kb_used_data":47624,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94297392,"statfs":{"total":96636764160,"available":96560529408,"internally_reserved":0,"allocated":48766976,"data_stored":48660540,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705709,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":74640,"kb_used_data":47816,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94297200,"statfs":{"total":96636764160,"available":96560332800,"internally_reserved":0,"allocated":48963584,"data_stored":48906874,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705708,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":79920,"kb_used_data":53096,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94291920,"statfs":{"total":96636764160,"available":96554926080,"internally_reserved":0,"allocated":54370304,"data_stored":53720806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":48545792,"data_stored":48182391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54038528,"data_stored":53597506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49553408,"data_stored":49405749,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":60440576,"data_stored":60219915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T01:56:01.325 INFO:tasks.rados.rados.0.smithi079.stdout:207: done (5 left) 2024-04-24T01:56:01.326 INFO:tasks.rados.rados.0.smithi079.stdout:208: delete oid 8 current snap is 15 2024-04-24T01:56:01.326 INFO:tasks.rados.rados.0.smithi079.stdout:201: expect (ObjNum 62 snap 2 seq_num 62) 2024-04-24T01:56:01.489 INFO:tasks.rados.rados.0.smithi079.stdout:200: finishing write tid 2 to smithi07929762-33 2024-04-24T01:56:01.489 INFO:tasks.rados.rados.0.smithi079.stdout:200: finishing write tid 3 to smithi07929762-33 2024-04-24T01:56:01.489 INFO:tasks.rados.rados.0.smithi079.stdout:200: finishing write tid 4 to smithi07929762-33 2024-04-24T01:56:01.489 INFO:tasks.rados.rados.0.smithi079.stdout:200: oid 33 updating version 0 to 40 2024-04-24T01:56:01.489 INFO:tasks.rados.rados.0.smithi079.stdout:200: oid 33 version 40 is already newer than 38 2024-04-24T01:56:01.489 INFO:tasks.rados.rados.0.smithi079.stdout:200: oid 33 version 40 is already newer than 40 2024-04-24T01:56:01.489 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 33 v 40 (ObjNum 124 snap 15 seq_num 124) dirty exists 2024-04-24T01:56:01.489 INFO:tasks.rados.rados.0.smithi079.stdout:200: left oid 33 (ObjNum 124 snap 15 seq_num 124) 2024-04-24T01:56:01.489 INFO:tasks.rados.rados.0.smithi079.stdout:200: done (5 left) 2024-04-24T01:56:01.489 INFO:tasks.rados.rados.0.smithi079.stdout:201: done (4 left) 2024-04-24T01:56:01.489 INFO:tasks.rados.rados.0.smithi079.stdout:208: done (3 left) 2024-04-24T01:56:01.490 INFO:tasks.rados.rados.0.smithi079.stdout:209: write (excl) oid 15 current snap is 15 2024-04-24T01:56:01.490 INFO:tasks.rados.rados.0.smithi079.stdout:205: finishing write tid 1 to smithi07929762-5 2024-04-24T01:56:01.490 INFO:tasks.rados.rados.0.smithi079.stdout:204: expect (ObjNum 31 snap 0 seq_num 31) 2024-04-24T01:56:01.581 INFO:tasks.rados.rados.0.smithi079.stdout:205: finishing write tid 2 to smithi07929762-5 2024-04-24T01:56:01.582 INFO:tasks.rados.rados.0.smithi079.stdout:206: expect (ObjNum 30 snap 0 seq_num 30) 2024-04-24T01:56:01.674 INFO:tasks.rados.rados.0.smithi079.stdout:205: finishing write tid 3 to smithi07929762-5 2024-04-24T01:56:01.675 INFO:tasks.rados.rados.0.smithi079.stdout:205: finishing write tid 4 to smithi07929762-5 2024-04-24T01:56:01.675 INFO:tasks.rados.rados.0.smithi079.stdout:205: oid 5 updating version 0 to 50 2024-04-24T01:56:01.675 INFO:tasks.rados.rados.0.smithi079.stdout:205: oid 5 updating version 50 to 52 2024-04-24T01:56:01.675 INFO:tasks.rados.rados.0.smithi079.stdout:205: oid 5 version 52 is already newer than 52 2024-04-24T01:56:01.675 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 5 v 52 (ObjNum 126 snap 15 seq_num 126) dirty exists 2024-04-24T01:56:01.675 INFO:tasks.rados.rados.0.smithi079.stdout:205: left oid 5 (ObjNum 126 snap 15 seq_num 126) 2024-04-24T01:56:01.675 INFO:tasks.rados.rados.0.smithi079.stdout:209: seq_num 128 ranges {551296=693520,2024048=127248} 2024-04-24T01:56:01.680 INFO:tasks.rados.rados.0.smithi079.stdout:209: writing smithi07929762-15 from 551296 to 1244816 tid 1 2024-04-24T01:56:01.686 INFO:tasks.rados.rados.0.smithi079.stdout:209: writing smithi07929762-15 from 2024048 to 2151296 tid 2 2024-04-24T01:56:01.686 INFO:tasks.rados.rados.0.smithi079.stdout:204: done (3 left) 2024-04-24T01:56:01.686 INFO:tasks.rados.rados.0.smithi079.stdout:205: done (2 left) 2024-04-24T01:56:01.686 INFO:tasks.rados.rados.0.smithi079.stdout:206: done (1 left) 2024-04-24T01:56:01.686 INFO:tasks.rados.rados.0.smithi079.stdout:210: read oid 31 snap -1 2024-04-24T01:56:01.687 INFO:tasks.rados.rados.0.smithi079.stdout:210: expect (ObjNum 30 snap 0 seq_num 30) 2024-04-24T01:56:01.687 INFO:tasks.rados.rados.0.smithi079.stdout:211: delete oid 3 current snap is 15 2024-04-24T01:56:01.737 INFO:tasks.rados.rados.0.smithi079.stdout:210: expect (ObjNum 30 snap 0 seq_num 30) 2024-04-24T01:56:01.841 INFO:tasks.rados.rados.0.smithi079.stdout:209: finishing write tid 1 to smithi07929762-15 2024-04-24T01:56:01.842 INFO:tasks.rados.rados.0.smithi079.stdout:209: finishing write tid 2 to smithi07929762-15 2024-04-24T01:56:01.842 INFO:tasks.rados.rados.0.smithi079.stdout:209: finishing write tid 3 to smithi07929762-15 2024-04-24T01:56:01.842 INFO:tasks.rados.rados.0.smithi079.stdout:209: finishing write tid 4 to smithi07929762-15 2024-04-24T01:56:01.842 INFO:tasks.rados.rados.0.smithi079.stdout:209: oid 15 updating version 0 to 46 2024-04-24T01:56:01.842 INFO:tasks.rados.rados.0.smithi079.stdout:209: oid 15 version 46 is already newer than 42 2024-04-24T01:56:01.842 INFO:tasks.rados.rados.0.smithi079.stdout:209: oid 15 version 46 is already newer than 46 2024-04-24T01:56:01.842 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 15 v 46 (ObjNum 128 snap 15 seq_num 128) dirty exists 2024-04-24T01:56:01.842 INFO:tasks.rados.rados.0.smithi079.stdout:209: left oid 15 (ObjNum 128 snap 15 seq_num 128) 2024-04-24T01:56:01.842 INFO:tasks.rados.rados.0.smithi079.stdout:209: done (2 left) 2024-04-24T01:56:01.842 INFO:tasks.rados.rados.0.smithi079.stdout:210: done (1 left) 2024-04-24T01:56:01.842 INFO:tasks.rados.rados.0.smithi079.stdout:211: done (0 left) 2024-04-24T01:56:01.842 INFO:tasks.rados.rados.0.smithi079.stdout:212: read oid 26 snap -1 2024-04-24T01:56:01.842 INFO:tasks.rados.rados.0.smithi079.stdout:212: expect (ObjNum 120 snap 10 seq_num 120) 2024-04-24T01:56:01.843 INFO:tasks.rados.rados.0.smithi079.stdout:213: write oid 47 current snap is 15 2024-04-24T01:56:01.843 INFO:tasks.rados.rados.0.smithi079.stdout:213: seq_num 130 ranges {584910=560371,1857365=327545} 2024-04-24T01:56:01.843 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T01:56:01.847 INFO:tasks.rados.rados.0.smithi079.stdout:213: writing smithi07929762-47 from 584910 to 1145281 tid 1 2024-04-24T01:56:01.854 INFO:tasks.rados.rados.0.smithi079.stdout:213: writing smithi07929762-47 from 1857365 to 2184910 tid 2 2024-04-24T01:56:01.854 INFO:tasks.rados.rados.0.smithi079.stdout:214: write oid 28 current snap is 15 2024-04-24T01:56:01.854 INFO:tasks.rados.rados.0.smithi079.stdout:214: seq_num 131 ranges {601717=635620,1799494=402223} 2024-04-24T01:56:01.861 INFO:tasks.rados.rados.0.smithi079.stdout:214: writing smithi07929762-28 from 601717 to 1237337 tid 1 2024-04-24T01:56:01.867 INFO:tasks.rados.rados.0.smithi079.stdout:214: writing smithi07929762-28 from 1799494 to 2201717 tid 2 2024-04-24T01:56:01.867 INFO:tasks.rados.rados.0.smithi079.stdout:215: snap_remove snap 11 2024-04-24T01:56:01.971 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:01.969+0000 7ff3ad99d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:01.975 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:01.974+0000 7ff3ad99d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:02.330 INFO:tasks.rados.rados.0.smithi079.stdout:215: done (3 left) 2024-04-24T01:56:02.331 INFO:tasks.rados.rados.0.smithi079.stdout:216: delete oid 13 current snap is 15 2024-04-24T01:56:02.331 INFO:tasks.rados.rados.0.smithi079.stdout:212: expect (ObjNum 120 snap 10 seq_num 120) 2024-04-24T01:56:02.497 INFO:tasks.rados.rados.0.smithi079.stdout:214: finishing write tid 1 to smithi07929762-28 2024-04-24T01:56:02.497 INFO:tasks.rados.rados.0.smithi079.stdout:213: finishing write tid 1 to smithi07929762-47 2024-04-24T01:56:02.497 INFO:tasks.rados.rados.0.smithi079.stdout:214: finishing write tid 2 to smithi07929762-28 2024-04-24T01:56:02.497 INFO:tasks.rados.rados.0.smithi079.stdout:213: finishing write tid 2 to smithi07929762-47 2024-04-24T01:56:02.497 INFO:tasks.rados.rados.0.smithi079.stdout:214: finishing write tid 3 to smithi07929762-28 2024-04-24T01:56:02.498 INFO:tasks.rados.rados.0.smithi079.stdout:214: finishing write tid 4 to smithi07929762-28 2024-04-24T01:56:02.498 INFO:tasks.rados.rados.0.smithi079.stdout:214: oid 28 updating version 0 to 48 2024-04-24T01:56:02.498 INFO:tasks.rados.rados.0.smithi079.stdout:214: oid 28 version 48 is already newer than 48 2024-04-24T01:56:02.498 INFO:tasks.rados.rados.0.smithi079.stdout:214: oid 28 version 48 is already newer than 48 2024-04-24T01:56:02.498 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 28 v 48 (ObjNum 131 snap 15 seq_num 131) dirty exists 2024-04-24T01:56:02.498 INFO:tasks.rados.rados.0.smithi079.stdout:214: left oid 28 (ObjNum 131 snap 15 seq_num 131) 2024-04-24T01:56:02.498 INFO:tasks.rados.rados.0.smithi079.stdout:213: finishing write tid 3 to smithi07929762-47 2024-04-24T01:56:02.498 INFO:tasks.rados.rados.0.smithi079.stdout:213: finishing write tid 4 to smithi07929762-47 2024-04-24T01:56:02.498 INFO:tasks.rados.rados.0.smithi079.stdout:213: oid 47 updating version 0 to 46 2024-04-24T01:56:02.498 INFO:tasks.rados.rados.0.smithi079.stdout:213: oid 47 version 46 is already newer than 46 2024-04-24T01:56:02.498 INFO:tasks.rados.rados.0.smithi079.stdout:213: oid 47 version 46 is already newer than 46 2024-04-24T01:56:02.498 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 47 v 46 (ObjNum 130 snap 15 seq_num 130) dirty exists 2024-04-24T01:56:02.498 INFO:tasks.rados.rados.0.smithi079.stdout:213: left oid 47 (ObjNum 130 snap 15 seq_num 130) 2024-04-24T01:56:02.498 INFO:tasks.rados.rados.0.smithi079.stdout:212: done (3 left) 2024-04-24T01:56:02.498 INFO:tasks.rados.rados.0.smithi079.stdout:213: done (2 left) 2024-04-24T01:56:02.498 INFO:tasks.rados.rados.0.smithi079.stdout:214: done (1 left) 2024-04-24T01:56:02.499 INFO:tasks.rados.rados.0.smithi079.stdout:216: done (0 left) 2024-04-24T01:56:02.499 INFO:tasks.rados.rados.0.smithi079.stdout:217: write (excl) oid 1 current snap is 15 2024-04-24T01:56:02.499 INFO:tasks.rados.rados.0.smithi079.stdout:217: seq_num 133 ranges {635331=786118,1916458=318873} 2024-04-24T01:56:02.504 INFO:tasks.rados.rados.0.smithi079.stdout:217: writing smithi07929762-1 from 635331 to 1421449 tid 1 2024-04-24T01:56:02.509 INFO:tasks.rados.rados.0.smithi079.stdout:217: writing smithi07929762-1 from 1916458 to 2235331 tid 2 2024-04-24T01:56:02.509 INFO:tasks.rados.rados.0.smithi079.stdout:218: delete oid 38 current snap is 15 2024-04-24T01:56:02.550 INFO:tasks.rados.rados.0.smithi079.stdout:218: done (1 left) 2024-04-24T01:56:02.550 INFO:tasks.rados.rados.0.smithi079.stdout:219: read oid 17 snap -1 2024-04-24T01:56:02.550 INFO:tasks.rados.rados.0.smithi079.stdout:219: expect deleted 2024-04-24T01:56:02.550 INFO:tasks.rados.rados.0.smithi079.stdout:220: write (excl) oid 16 current snap is 15 2024-04-24T01:56:02.550 INFO:tasks.rados.rados.0.smithi079.stdout:220: seq_num 135 ranges {668945=536616,1633422=491363} 2024-04-24T01:56:02.557 INFO:tasks.rados.rados.0.smithi079.stdout:220: writing smithi07929762-16 from 668945 to 1205561 tid 1 2024-04-24T01:56:02.564 INFO:tasks.rados.rados.0.smithi079.stdout:220: writing smithi07929762-16 from 1633422 to 2124785 tid 2 2024-04-24T01:56:02.564 INFO:tasks.rados.rados.0.smithi079.stdout:221: snap_create 2024-04-24T01:56:02.565 INFO:tasks.rados.rados.0.smithi079.stdout:217: finishing write tid 1 to smithi07929762-1 2024-04-24T01:56:02.572 INFO:tasks.rados.rados.0.smithi079.stdout:217: finishing write tid 2 to smithi07929762-1 2024-04-24T01:56:02.576 INFO:tasks.rados.rados.0.smithi079.stdout:217: finishing write tid 3 to smithi07929762-1 2024-04-24T01:56:02.579 INFO:tasks.rados.rados.0.smithi079.stdout:217: finishing write tid 4 to smithi07929762-1 2024-04-24T01:56:02.579 INFO:tasks.rados.rados.0.smithi079.stdout:217: oid 1 updating version 0 to 59 2024-04-24T01:56:02.579 INFO:tasks.rados.rados.0.smithi079.stdout:217: oid 1 updating version 59 to 61 2024-04-24T01:56:02.579 INFO:tasks.rados.rados.0.smithi079.stdout:217: oid 1 version 61 is already newer than 61 2024-04-24T01:56:02.579 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 1 v 61 (ObjNum 133 snap 15 seq_num 133) dirty exists 2024-04-24T01:56:02.580 INFO:tasks.rados.rados.0.smithi079.stdout:217: left oid 1 (ObjNum 133 snap 15 seq_num 133) 2024-04-24T01:56:02.605 INFO:tasks.rados.rados.0.smithi079.stdout:220: finishing write tid 1 to smithi07929762-16 2024-04-24T01:56:02.620 INFO:tasks.rados.rados.0.smithi079.stdout:220: finishing write tid 2 to smithi07929762-16 2024-04-24T01:56:02.623 INFO:tasks.rados.rados.0.smithi079.stdout:220: finishing write tid 3 to smithi07929762-16 2024-04-24T01:56:02.625 INFO:tasks.rados.rados.0.smithi079.stdout:220: finishing write tid 4 to smithi07929762-16 2024-04-24T01:56:02.625 INFO:tasks.rados.rados.0.smithi079.stdout:220: oid 16 updating version 0 to 63 2024-04-24T01:56:02.625 INFO:tasks.rados.rados.0.smithi079.stdout:220: oid 16 updating version 63 to 65 2024-04-24T01:56:02.625 INFO:tasks.rados.rados.0.smithi079.stdout:220: oid 16 version 65 is already newer than 65 2024-04-24T01:56:02.625 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 16 v 65 (ObjNum 135 snap 15 seq_num 135) dirty exists 2024-04-24T01:56:02.625 INFO:tasks.rados.rados.0.smithi079.stdout:220: left oid 16 (ObjNum 135 snap 15 seq_num 135) 2024-04-24T01:56:03.334 INFO:tasks.rados.rados.0.smithi079.stdout:217: done (3 left) 2024-04-24T01:56:03.334 INFO:tasks.rados.rados.0.smithi079.stdout:219: done (2 left) 2024-04-24T01:56:03.334 INFO:tasks.rados.rados.0.smithi079.stdout:220: done (1 left) 2024-04-24T01:56:03.334 INFO:tasks.rados.rados.0.smithi079.stdout:221: done (0 left) 2024-04-24T01:56:03.334 INFO:tasks.rados.rados.0.smithi079.stdout:222: snap_create 2024-04-24T01:56:04.073 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T01:56:04.187 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:04.185+0000 7f7d1de6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:04.191 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:04.191+0000 7f7d1de6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:04.335 INFO:tasks.rados.rados.0.smithi079.stdout:222: done (0 left) 2024-04-24T01:56:04.335 INFO:tasks.rados.rados.0.smithi079.stdout:223: write (excl) oid 45 current snap is 17 2024-04-24T01:56:04.335 INFO:tasks.rados.rados.0.smithi079.stdout:223: seq_num 136 ranges {685752=611865,1975551=310201} 2024-04-24T01:56:04.342 INFO:tasks.rados.rados.0.smithi079.stdout:223: writing smithi07929762-45 from 685752 to 1297617 tid 1 2024-04-24T01:56:04.348 INFO:tasks.rados.rados.0.smithi079.stdout:223: writing smithi07929762-45 from 1975551 to 2285752 tid 2 2024-04-24T01:56:04.348 INFO:tasks.rados.rados.0.smithi079.stdout:224: write oid 25 current snap is 17 2024-04-24T01:56:04.349 INFO:tasks.rados.rados.0.smithi079.stdout:224: seq_num 137 ranges {702559=403467,1750386=495032} 2024-04-24T01:56:04.356 INFO:tasks.rados.rados.0.smithi079.stdout:224: writing smithi07929762-25 from 702559 to 1106026 tid 1 2024-04-24T01:56:04.362 INFO:tasks.rados.rados.0.smithi079.stdout:224: writing smithi07929762-25 from 1750386 to 2245418 tid 2 2024-04-24T01:56:04.362 INFO:tasks.rados.rados.0.smithi079.stdout:225: write (excl) oid 19 current snap is 17 2024-04-24T01:56:04.362 INFO:tasks.rados.rados.0.smithi079.stdout:225: seq_num 138 ranges {719366=478716,1808868=510498} 2024-04-24T01:56:04.369 INFO:tasks.rados.rados.0.smithi079.stdout:225: writing smithi07929762-19 from 719366 to 1198082 tid 1 2024-04-24T01:56:04.376 INFO:tasks.rados.rados.0.smithi079.stdout:225: writing smithi07929762-19 from 1808868 to 2319366 tid 2 2024-04-24T01:56:04.376 INFO:tasks.rados.rados.0.smithi079.stdout:226: read oid 44 snap -1 2024-04-24T01:56:04.376 INFO:tasks.rados.rados.0.smithi079.stdout:226: expect (ObjNum 105 snap 9 seq_num 105) 2024-04-24T01:56:04.376 INFO:tasks.rados.rados.0.smithi079.stdout:227: read oid 32 snap -1 2024-04-24T01:56:04.376 INFO:tasks.rados.rados.0.smithi079.stdout:227: expect (ObjNum 31 snap 0 seq_num 31) 2024-04-24T01:56:04.376 INFO:tasks.rados.rados.0.smithi079.stdout:228: read oid 36 snap -1 2024-04-24T01:56:04.376 INFO:tasks.rados.rados.0.smithi079.stdout:228: expect deleted 2024-04-24T01:56:04.376 INFO:tasks.rados.rados.0.smithi079.stdout:229: read oid 6 snap -1 2024-04-24T01:56:04.377 INFO:tasks.rados.rados.0.smithi079.stdout:229: expect (ObjNum 66 snap 3 seq_num 66) 2024-04-24T01:56:04.377 INFO:tasks.rados.rados.0.smithi079.stdout:230: snap_remove snap 15 2024-04-24T01:56:05.344 INFO:tasks.rados.rados.0.smithi079.stdout:230: done (7 left) 2024-04-24T01:56:05.344 INFO:tasks.rados.rados.0.smithi079.stdout:231: read oid 31 snap -1 2024-04-24T01:56:05.345 INFO:tasks.rados.rados.0.smithi079.stdout:231: expect (ObjNum 30 snap 0 seq_num 30) 2024-04-24T01:56:05.345 INFO:tasks.rados.rados.0.smithi079.stdout:232: write oid 2 current snap is 17 2024-04-24T01:56:05.345 INFO:tasks.rados.rados.0.smithi079.stdout:232: seq_num 139 ranges {736173=553965,1867350=468823} 2024-04-24T01:56:05.351 INFO:tasks.rados.rados.0.smithi079.stdout:232: writing smithi07929762-2 from 736173 to 1290138 tid 1 2024-04-24T01:56:05.358 INFO:tasks.rados.rados.0.smithi079.stdout:232: writing smithi07929762-2 from 1867350 to 2336173 tid 2 2024-04-24T01:56:05.358 INFO:tasks.rados.rados.0.smithi079.stdout:233: read oid 29 snap -1 2024-04-24T01:56:05.358 INFO:tasks.rados.rados.0.smithi079.stdout:233: expect (ObjNum 91 snap 7 seq_num 91) 2024-04-24T01:56:05.358 INFO:tasks.rados.rados.0.smithi079.stdout:234: write oid 14 current snap is 17 2024-04-24T01:56:05.358 INFO:tasks.rados.rados.0.smithi079.stdout:234: seq_num 140 ranges {752980=629214,1809479=543501} 2024-04-24T01:56:05.366 INFO:tasks.rados.rados.0.smithi079.stdout:234: writing smithi07929762-14 from 752980 to 1382194 tid 1 2024-04-24T01:56:05.372 INFO:tasks.rados.rados.0.smithi079.stdout:234: writing smithi07929762-14 from 1809479 to 2352980 tid 2 2024-04-24T01:56:05.372 INFO:tasks.rados.rados.0.smithi079.stdout:235: delete oid 16 current snap is 17 2024-04-24T01:56:05.372 INFO:tasks.rados.rados.0.smithi079.stdout:229: expect (ObjNum 66 snap 3 seq_num 66) 2024-04-24T01:56:05.550 INFO:tasks.rados.rados.0.smithi079.stdout:223: finishing write tid 1 to smithi07929762-45 2024-04-24T01:56:05.551 INFO:tasks.rados.rados.0.smithi079.stdout:227: expect (ObjNum 31 snap 0 seq_num 31) 2024-04-24T01:56:05.644 INFO:tasks.rados.rados.0.smithi079.stdout:224: finishing write tid 1 to smithi07929762-25 2024-04-24T01:56:05.644 INFO:tasks.rados.rados.0.smithi079.stdout:226: expect (ObjNum 105 snap 9 seq_num 105) 2024-04-24T01:56:05.871 INFO:tasks.rados.rados.0.smithi079.stdout:225: finishing write tid 1 to smithi07929762-19 2024-04-24T01:56:05.871 INFO:tasks.rados.rados.0.smithi079.stdout:224: finishing write tid 2 to smithi07929762-25 2024-04-24T01:56:05.871 INFO:tasks.rados.rados.0.smithi079.stdout:223: finishing write tid 2 to smithi07929762-45 2024-04-24T01:56:05.871 INFO:tasks.rados.rados.0.smithi079.stdout:224: finishing write tid 3 to smithi07929762-25 2024-04-24T01:56:05.871 INFO:tasks.rados.rados.0.smithi079.stdout:224: finishing write tid 4 to smithi07929762-25 2024-04-24T01:56:05.871 INFO:tasks.rados.rados.0.smithi079.stdout:224: oid 25 updating version 0 to 54 2024-04-24T01:56:05.871 INFO:tasks.rados.rados.0.smithi079.stdout:224: oid 25 version 54 is already newer than 54 2024-04-24T01:56:05.871 INFO:tasks.rados.rados.0.smithi079.stdout:224: oid 25 version 54 is already newer than 54 2024-04-24T01:56:05.871 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 25 v 54 (ObjNum 137 snap 17 seq_num 137) dirty exists 2024-04-24T01:56:05.872 INFO:tasks.rados.rados.0.smithi079.stdout:224: left oid 25 (ObjNum 137 snap 17 seq_num 137) 2024-04-24T01:56:05.872 INFO:tasks.rados.rados.0.smithi079.stdout:225: finishing write tid 2 to smithi07929762-19 2024-04-24T01:56:05.872 INFO:tasks.rados.rados.0.smithi079.stdout:223: finishing write tid 3 to smithi07929762-45 2024-04-24T01:56:05.872 INFO:tasks.rados.rados.0.smithi079.stdout:225: finishing write tid 3 to smithi07929762-19 2024-04-24T01:56:05.872 INFO:tasks.rados.rados.0.smithi079.stdout:223: finishing write tid 4 to smithi07929762-45 2024-04-24T01:56:05.872 INFO:tasks.rados.rados.0.smithi079.stdout:223: oid 45 updating version 0 to 54 2024-04-24T01:56:05.872 INFO:tasks.rados.rados.0.smithi079.stdout:223: oid 45 version 54 is already newer than 52 2024-04-24T01:56:05.872 INFO:tasks.rados.rados.0.smithi079.stdout:223: oid 45 version 54 is already newer than 54 2024-04-24T01:56:05.872 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 45 v 54 (ObjNum 136 snap 17 seq_num 136) dirty exists 2024-04-24T01:56:05.872 INFO:tasks.rados.rados.0.smithi079.stdout:223: left oid 45 (ObjNum 136 snap 17 seq_num 136) 2024-04-24T01:56:05.872 INFO:tasks.rados.rados.0.smithi079.stdout:223: done (11 left) 2024-04-24T01:56:05.872 INFO:tasks.rados.rados.0.smithi079.stdout:224: done (10 left) 2024-04-24T01:56:05.872 INFO:tasks.rados.rados.0.smithi079.stdout:226: done (9 left) 2024-04-24T01:56:05.872 INFO:tasks.rados.rados.0.smithi079.stdout:227: done (8 left) 2024-04-24T01:56:05.872 INFO:tasks.rados.rados.0.smithi079.stdout:228: done (7 left) 2024-04-24T01:56:05.873 INFO:tasks.rados.rados.0.smithi079.stdout:229: done (6 left) 2024-04-24T01:56:05.873 INFO:tasks.rados.rados.0.smithi079.stdout:235: done (5 left) 2024-04-24T01:56:05.873 INFO:tasks.rados.rados.0.smithi079.stdout:236: read oid 28 snap -1 2024-04-24T01:56:05.873 INFO:tasks.rados.rados.0.smithi079.stdout:236: expect (ObjNum 131 snap 15 seq_num 131) 2024-04-24T01:56:05.873 INFO:tasks.rados.rados.0.smithi079.stdout:237: delete oid 4 current snap is 17 2024-04-24T01:56:05.873 INFO:tasks.rados.rados.0.smithi079.stdout:225: finishing write tid 4 to smithi07929762-19 2024-04-24T01:56:05.873 INFO:tasks.rados.rados.0.smithi079.stdout:225: oid 19 updating version 0 to 32 2024-04-24T01:56:05.873 INFO:tasks.rados.rados.0.smithi079.stdout:225: oid 19 version 32 is already newer than 30 2024-04-24T01:56:05.873 INFO:tasks.rados.rados.0.smithi079.stdout:225: oid 19 version 32 is already newer than 32 2024-04-24T01:56:05.873 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 19 v 32 (ObjNum 138 snap 17 seq_num 138) dirty exists 2024-04-24T01:56:05.873 INFO:tasks.rados.rados.0.smithi079.stdout:225: left oid 19 (ObjNum 138 snap 17 seq_num 138) 2024-04-24T01:56:05.873 INFO:tasks.rados.rados.0.smithi079.stdout:232: finishing write tid 1 to smithi07929762-2 2024-04-24T01:56:05.873 INFO:tasks.rados.rados.0.smithi079.stdout:233: expect (ObjNum 91 snap 7 seq_num 91) 2024-04-24T01:56:06.048 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-04-24T01:56:06.048 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-04-24T01:56:06.049 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-04-24T01:56:06.049 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-04-24T01:56:06.088 INFO:tasks.rados.rados.0.smithi079.stdout:231: expect (ObjNum 30 snap 0 seq_num 30) 2024-04-24T01:56:06.159 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:06.153+0000 7f6af226d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:06.162 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:06.161+0000 7f6af226d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:06.179 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T01:56:06.179 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T01:56:06.179 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T01:56:06.179 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T01:56:06.179 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T01:56:06.179 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T01:56:06.179 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T01:56:06.179 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T01:56:06.179 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T01:56:06.179 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T01:56:06.179 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T01:56:06.179 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T01:56:06.182 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T01:56:06.183 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T01:56:06.183 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-04-24T01:56:06.187 INFO:tasks.rados.rados.0.smithi079.stdout:232: finishing write tid 2 to smithi07929762-2 2024-04-24T01:56:06.187 INFO:tasks.rados.rados.0.smithi079.stdout:232: finishing write tid 3 to smithi07929762-2 2024-04-24T01:56:06.187 INFO:tasks.rados.rados.0.smithi079.stdout:232: finishing write tid 4 to smithi07929762-2 2024-04-24T01:56:06.187 INFO:tasks.rados.rados.0.smithi079.stdout:232: oid 2 updating version 0 to 19 2024-04-24T01:56:06.187 INFO:tasks.rados.rados.0.smithi079.stdout:232: oid 2 version 19 is already newer than 19 2024-04-24T01:56:06.187 INFO:tasks.rados.rados.0.smithi079.stdout:232: oid 2 version 19 is already newer than 19 2024-04-24T01:56:06.188 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 2 v 19 (ObjNum 139 snap 17 seq_num 139) dirty exists 2024-04-24T01:56:06.188 INFO:tasks.rados.rados.0.smithi079.stdout:232: left oid 2 (ObjNum 139 snap 17 seq_num 139) 2024-04-24T01:56:06.188 INFO:tasks.rados.rados.0.smithi079.stdout:234: finishing write tid 1 to smithi07929762-14 2024-04-24T01:56:06.188 INFO:tasks.rados.rados.0.smithi079.stdout:234: finishing write tid 2 to smithi07929762-14 2024-04-24T01:56:06.188 INFO:tasks.rados.rados.0.smithi079.stdout:234: finishing write tid 3 to smithi07929762-14 2024-04-24T01:56:06.188 INFO:tasks.rados.rados.0.smithi079.stdout:225: done (6 left) 2024-04-24T01:56:06.188 INFO:tasks.rados.rados.0.smithi079.stdout:231: done (5 left) 2024-04-24T01:56:06.188 INFO:tasks.rados.rados.0.smithi079.stdout:232: done (4 left) 2024-04-24T01:56:06.188 INFO:tasks.rados.rados.0.smithi079.stdout:233: done (3 left) 2024-04-24T01:56:06.188 INFO:tasks.rados.rados.0.smithi079.stdout:237: done (2 left) 2024-04-24T01:56:06.188 INFO:tasks.rados.rados.0.smithi079.stdout:238: snap_remove snap 12 2024-04-24T01:56:06.290 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T01:56:06.310 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:06.304+0000 7f3ad6e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:06.314 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:06.314+0000 7f3ad6e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:06.333 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T01:56:06.333 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T01:56:06.333 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T01:56:06.333 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T01:56:06.333 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T01:56:06.333 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T01:56:06.333 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T01:56:06.333 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T01:56:06.334 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T01:56:06.334 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T01:56:06.334 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T01:56:06.334 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T01:56:06.336 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T01:56:06.336 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T01:56:06.336 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-04-24T01:56:06.343 INFO:tasks.rados.rados.0.smithi079.stdout:238: done (2 left) 2024-04-24T01:56:06.343 INFO:tasks.rados.rados.0.smithi079.stdout:239: read oid 26 snap -1 2024-04-24T01:56:06.344 INFO:tasks.rados.rados.0.smithi079.stdout:239: expect (ObjNum 120 snap 10 seq_num 120) 2024-04-24T01:56:06.344 INFO:tasks.rados.rados.0.smithi079.stdout:240: snap_remove snap 16 2024-04-24T01:56:06.425 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:06.422+0000 7f3b52d5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:06.430 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:06.430+0000 7f3b52d5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:06.482 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:06.474+0000 7f848a6ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:06.486 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:06.486+0000 7f848a6ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:06.504 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T01:56:06.504 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T01:56:06.504 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T01:56:06.505 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T01:56:06.505 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T01:56:06.505 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T01:56:06.505 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T01:56:06.505 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T01:56:06.505 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T01:56:06.505 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T01:56:06.505 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T01:56:06.505 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T01:56:06.507 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T01:56:06.508 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T01:56:06.508 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-04-24T01:56:06.617 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:06.612+0000 7f8b7762d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:06.621 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:06.620+0000 7f8b7762d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:06.637 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T01:56:06.637 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T01:56:06.637 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T01:56:06.637 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T01:56:06.637 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T01:56:06.637 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T01:56:06.637 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T01:56:06.637 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T01:56:06.637 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T01:56:06.637 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T01:56:06.637 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T01:56:06.637 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T01:56:06.640 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T01:56:06.640 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T01:56:07.350 INFO:tasks.rados.rados.0.smithi079.stdout:240: done (3 left) 2024-04-24T01:56:07.350 INFO:tasks.rados.rados.0.smithi079.stdout:241: read oid 25 snap -1 2024-04-24T01:56:07.350 INFO:tasks.rados.rados.0.smithi079.stdout:241: expect (ObjNum 137 snap 17 seq_num 137) 2024-04-24T01:56:07.350 INFO:tasks.rados.rados.0.smithi079.stdout:242: read oid 3 snap -1 2024-04-24T01:56:07.350 INFO:tasks.rados.rados.0.smithi079.stdout:242: expect deleted 2024-04-24T01:56:07.350 INFO:tasks.rados.rados.0.smithi079.stdout:243: snap_create 2024-04-24T01:56:07.350 INFO:tasks.rados.rados.0.smithi079.stdout:234: finishing write tid 4 to smithi07929762-14 2024-04-24T01:56:07.350 INFO:tasks.rados.rados.0.smithi079.stdout:234: oid 14 updating version 0 to 46 2024-04-24T01:56:07.350 INFO:tasks.rados.rados.0.smithi079.stdout:234: oid 14 version 46 is already newer than 46 2024-04-24T01:56:07.351 INFO:tasks.rados.rados.0.smithi079.stdout:234: oid 14 version 46 is already newer than 46 2024-04-24T01:56:07.351 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 14 v 46 (ObjNum 140 snap 17 seq_num 140) dirty exists 2024-04-24T01:56:07.351 INFO:tasks.rados.rados.0.smithi079.stdout:234: left oid 14 (ObjNum 140 snap 17 seq_num 140) 2024-04-24T01:56:07.351 INFO:tasks.rados.rados.0.smithi079.stdout:236: expect (ObjNum 131 snap 15 seq_num 131) 2024-04-24T01:56:07.508 INFO:tasks.rados.rados.0.smithi079.stdout:239: expect (ObjNum 120 snap 10 seq_num 120) 2024-04-24T01:56:07.650 INFO:tasks.rados.rados.0.smithi079.stdout:241: expect (ObjNum 137 snap 17 seq_num 137) 2024-04-24T01:56:08.350 INFO:tasks.rados.rados.0.smithi079.stdout:234: done (5 left) 2024-04-24T01:56:08.350 INFO:tasks.rados.rados.0.smithi079.stdout:236: done (4 left) 2024-04-24T01:56:08.350 INFO:tasks.rados.rados.0.smithi079.stdout:239: done (3 left) 2024-04-24T01:56:08.350 INFO:tasks.rados.rados.0.smithi079.stdout:241: done (2 left) 2024-04-24T01:56:08.350 INFO:tasks.rados.rados.0.smithi079.stdout:242: done (1 left) 2024-04-24T01:56:08.350 INFO:tasks.rados.rados.0.smithi079.stdout:243: done (0 left) 2024-04-24T01:56:08.350 INFO:tasks.rados.rados.0.smithi079.stdout:244: write (excl) oid 49 current snap is 18 2024-04-24T01:56:08.350 INFO:tasks.rados.rados.0.smithi079.stdout:244: seq_num 143 ranges {403401=454961,1584925=506039} 2024-04-24T01:56:08.354 INFO:tasks.rados.rados.0.smithi079.stdout:244: writing smithi07929762-49 from 403401 to 858362 tid 1 2024-04-24T01:56:08.361 INFO:tasks.rados.rados.0.smithi079.stdout:244: writing smithi07929762-49 from 1584925 to 2090964 tid 2 2024-04-24T01:56:08.361 INFO:tasks.rados.rados.0.smithi079.stdout:245: snap_create 2024-04-24T01:56:08.406 INFO:tasks.rados.rados.0.smithi079.stdout:244: finishing write tid 1 to smithi07929762-49 2024-04-24T01:56:08.420 INFO:tasks.rados.rados.0.smithi079.stdout:244: finishing write tid 2 to smithi07929762-49 2024-04-24T01:56:08.423 INFO:tasks.rados.rados.0.smithi079.stdout:244: finishing write tid 3 to smithi07929762-49 2024-04-24T01:56:08.429 INFO:tasks.rados.rados.0.smithi079.stdout:244: finishing write tid 4 to smithi07929762-49 2024-04-24T01:56:08.429 INFO:tasks.rados.rados.0.smithi079.stdout:244: oid 49 updating version 0 to 78 2024-04-24T01:56:08.429 INFO:tasks.rados.rados.0.smithi079.stdout:244: oid 49 version 78 is already newer than 78 2024-04-24T01:56:08.429 INFO:tasks.rados.rados.0.smithi079.stdout:244: oid 49 version 78 is already newer than 76 2024-04-24T01:56:08.429 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 49 v 78 (ObjNum 143 snap 18 seq_num 143) dirty exists 2024-04-24T01:56:08.430 INFO:tasks.rados.rados.0.smithi079.stdout:244: left oid 49 (ObjNum 143 snap 18 seq_num 143) 2024-04-24T01:56:08.560 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T01:56:08.664 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:08.662+0000 7f9d1646d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:08.668 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:08.667+0000 7f9d1646d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:09.358 INFO:tasks.rados.rados.0.smithi079.stdout:244: done (1 left) 2024-04-24T01:56:09.358 INFO:tasks.rados.rados.0.smithi079.stdout:245: done (0 left) 2024-04-24T01:56:09.359 INFO:tasks.rados.rados.0.smithi079.stdout:246: read oid 49 snap -1 2024-04-24T01:56:09.359 INFO:tasks.rados.rados.0.smithi079.stdout:246: expect (ObjNum 143 snap 18 seq_num 143) 2024-04-24T01:56:09.359 INFO:tasks.rados.rados.0.smithi079.stdout:247: write oid 37 current snap is 19 2024-04-24T01:56:09.359 INFO:tasks.rados.rados.0.smithi079.stdout:247: seq_num 144 ranges {420208=530210,1527054=766050} 2024-04-24T01:56:09.363 INFO:tasks.rados.rados.0.smithi079.stdout:247: writing smithi07929762-37 from 420208 to 950418 tid 1 2024-04-24T01:56:09.372 INFO:tasks.rados.rados.0.smithi079.stdout:247: writing smithi07929762-37 from 1527054 to 2293104 tid 2 2024-04-24T01:56:09.372 INFO:tasks.rados.rados.0.smithi079.stdout:248: write oid 48 current snap is 19 2024-04-24T01:56:09.372 INFO:tasks.rados.rados.0.smithi079.stdout:248: seq_num 145 ranges {437015=721812,1701889=509708} 2024-04-24T01:56:09.379 INFO:tasks.rados.rados.0.smithi079.stdout:248: writing smithi07929762-48 from 437015 to 1158827 tid 1 2024-04-24T01:56:09.387 INFO:tasks.rados.rados.0.smithi079.stdout:248: writing smithi07929762-48 from 1701889 to 2211597 tid 2 2024-04-24T01:56:09.388 INFO:tasks.rados.rados.0.smithi079.stdout:249: snap_create 2024-04-24T01:56:09.461 INFO:tasks.rados.rados.0.smithi079.stdout:246: expect (ObjNum 143 snap 18 seq_num 143) 2024-04-24T01:56:09.563 INFO:tasks.rados.rados.0.smithi079.stdout:248: finishing write tid 1 to smithi07929762-48 2024-04-24T01:56:09.563 INFO:tasks.rados.rados.0.smithi079.stdout:248: finishing write tid 2 to smithi07929762-48 2024-04-24T01:56:09.563 INFO:tasks.rados.rados.0.smithi079.stdout:248: finishing write tid 3 to smithi07929762-48 2024-04-24T01:56:09.563 INFO:tasks.rados.rados.0.smithi079.stdout:248: finishing write tid 4 to smithi07929762-48 2024-04-24T01:56:09.563 INFO:tasks.rados.rados.0.smithi079.stdout:248: oid 48 updating version 0 to 10 2024-04-24T01:56:09.563 INFO:tasks.rados.rados.0.smithi079.stdout:248: oid 48 version 10 is already newer than 10 2024-04-24T01:56:09.563 INFO:tasks.rados.rados.0.smithi079.stdout:248: oid 48 version 10 is already newer than 10 2024-04-24T01:56:09.563 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 48 v 10 (ObjNum 145 snap 19 seq_num 145) dirty exists 2024-04-24T01:56:09.563 INFO:tasks.rados.rados.0.smithi079.stdout:248: left oid 48 (ObjNum 145 snap 19 seq_num 145) 2024-04-24T01:56:09.563 INFO:tasks.rados.rados.0.smithi079.stdout:247: finishing write tid 1 to smithi07929762-37 2024-04-24T01:56:09.563 INFO:tasks.rados.rados.0.smithi079.stdout:247: finishing write tid 2 to smithi07929762-37 2024-04-24T01:56:09.563 INFO:tasks.rados.rados.0.smithi079.stdout:247: finishing write tid 3 to smithi07929762-37 2024-04-24T01:56:09.563 INFO:tasks.rados.rados.0.smithi079.stdout:247: finishing write tid 4 to smithi07929762-37 2024-04-24T01:56:09.564 INFO:tasks.rados.rados.0.smithi079.stdout:247: oid 37 updating version 0 to 82 2024-04-24T01:56:09.564 INFO:tasks.rados.rados.0.smithi079.stdout:247: oid 37 version 82 is already newer than 82 2024-04-24T01:56:09.564 INFO:tasks.rados.rados.0.smithi079.stdout:247: oid 37 version 82 is already newer than 82 2024-04-24T01:56:09.564 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 37 v 82 (ObjNum 144 snap 19 seq_num 144) dirty exists 2024-04-24T01:56:09.564 INFO:tasks.rados.rados.0.smithi079.stdout:247: left oid 37 (ObjNum 144 snap 19 seq_num 144) 2024-04-24T01:56:10.358 INFO:tasks.rados.rados.0.smithi079.stdout:246: done (3 left) 2024-04-24T01:56:10.358 INFO:tasks.rados.rados.0.smithi079.stdout:247: done (2 left) 2024-04-24T01:56:10.358 INFO:tasks.rados.rados.0.smithi079.stdout:248: done (1 left) 2024-04-24T01:56:10.358 INFO:tasks.rados.rados.0.smithi079.stdout:249: done (0 left) 2024-04-24T01:56:10.358 INFO:tasks.rados.rados.0.smithi079.stdout:250: read oid 44 snap -1 2024-04-24T01:56:10.358 INFO:tasks.rados.rados.0.smithi079.stdout:250: expect (ObjNum 105 snap 9 seq_num 105) 2024-04-24T01:56:10.358 INFO:tasks.rados.rados.0.smithi079.stdout:251: write (excl) oid 32 current snap is 20 2024-04-24T01:56:10.358 INFO:tasks.rados.rados.0.smithi079.stdout:251: seq_num 146 ranges {453822=797061,1760371=653366} 2024-04-24T01:56:10.366 INFO:tasks.rados.rados.0.smithi079.stdout:251: writing smithi07929762-32 from 453822 to 1250883 tid 1 2024-04-24T01:56:10.372 INFO:tasks.rados.rados.0.smithi079.stdout:251: writing smithi07929762-32 from 1760371 to 2413737 tid 2 2024-04-24T01:56:10.373 INFO:tasks.rados.rados.0.smithi079.stdout:252: write oid 10 current snap is 20 2024-04-24T01:56:10.373 INFO:tasks.rados.rados.0.smithi079.stdout:252: seq_num 147 ranges {470629=472310,1418853=513377} 2024-04-24T01:56:10.380 INFO:tasks.rados.rados.0.smithi079.stdout:252: writing smithi07929762-10 from 470629 to 942939 tid 1 2024-04-24T01:56:10.387 INFO:tasks.rados.rados.0.smithi079.stdout:252: writing smithi07929762-10 from 1418853 to 1932230 tid 2 2024-04-24T01:56:10.387 INFO:tasks.rados.rados.0.smithi079.stdout:253: delete oid 46 current snap is 20 2024-04-24T01:56:10.409 INFO:tasks.rados.rados.0.smithi079.stdout:253: done (3 left) 2024-04-24T01:56:10.409 INFO:tasks.rados.rados.0.smithi079.stdout:254: snap_remove snap 19 2024-04-24T01:56:10.766 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T01:56:10.870 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:10.868+0000 7f52048ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:10.874 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:10.873+0000 7f52048ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:11.371 INFO:tasks.rados.rados.0.smithi079.stdout:254: done (3 left) 2024-04-24T01:56:11.371 INFO:tasks.rados.rados.0.smithi079.stdout:255: delete oid 43 current snap is 20 2024-04-24T01:56:11.371 INFO:tasks.rados.rados.0.smithi079.stdout:250: expect (ObjNum 105 snap 9 seq_num 105) 2024-04-24T01:56:11.598 INFO:tasks.rados.rados.0.smithi079.stdout:252: finishing write tid 1 to smithi07929762-10 2024-04-24T01:56:11.598 INFO:tasks.rados.rados.0.smithi079.stdout:251: finishing write tid 1 to smithi07929762-32 2024-04-24T01:56:11.598 INFO:tasks.rados.rados.0.smithi079.stdout:252: finishing write tid 2 to smithi07929762-10 2024-04-24T01:56:11.598 INFO:tasks.rados.rados.0.smithi079.stdout:252: finishing write tid 3 to smithi07929762-10 2024-04-24T01:56:11.598 INFO:tasks.rados.rados.0.smithi079.stdout:252: finishing write tid 4 to smithi07929762-10 2024-04-24T01:56:11.598 INFO:tasks.rados.rados.0.smithi079.stdout:252: oid 10 updating version 0 to 65 2024-04-24T01:56:11.598 INFO:tasks.rados.rados.0.smithi079.stdout:252: oid 10 version 65 is already newer than 65 2024-04-24T01:56:11.598 INFO:tasks.rados.rados.0.smithi079.stdout:252: oid 10 version 65 is already newer than 65 2024-04-24T01:56:11.598 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 10 v 65 (ObjNum 147 snap 20 seq_num 147) dirty exists 2024-04-24T01:56:11.598 INFO:tasks.rados.rados.0.smithi079.stdout:252: left oid 10 (ObjNum 147 snap 20 seq_num 147) 2024-04-24T01:56:11.599 INFO:tasks.rados.rados.0.smithi079.stdout:251: finishing write tid 2 to smithi07929762-32 2024-04-24T01:56:11.599 INFO:tasks.rados.rados.0.smithi079.stdout:251: finishing write tid 3 to smithi07929762-32 2024-04-24T01:56:11.599 INFO:tasks.rados.rados.0.smithi079.stdout:251: finishing write tid 4 to smithi07929762-32 2024-04-24T01:56:11.599 INFO:tasks.rados.rados.0.smithi079.stdout:251: oid 32 updating version 0 to 14 2024-04-24T01:56:11.599 INFO:tasks.rados.rados.0.smithi079.stdout:251: oid 32 version 14 is already newer than 14 2024-04-24T01:56:11.599 INFO:tasks.rados.rados.0.smithi079.stdout:251: oid 32 version 14 is already newer than 12 2024-04-24T01:56:11.599 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 32 v 14 (ObjNum 146 snap 20 seq_num 146) dirty exists 2024-04-24T01:56:11.599 INFO:tasks.rados.rados.0.smithi079.stdout:251: left oid 32 (ObjNum 146 snap 20 seq_num 146) 2024-04-24T01:56:11.599 INFO:tasks.rados.rados.0.smithi079.stdout:250: done (3 left) 2024-04-24T01:56:11.599 INFO:tasks.rados.rados.0.smithi079.stdout:251: done (2 left) 2024-04-24T01:56:11.599 INFO:tasks.rados.rados.0.smithi079.stdout:252: done (1 left) 2024-04-24T01:56:11.599 INFO:tasks.rados.rados.0.smithi079.stdout:255: done (0 left) 2024-04-24T01:56:11.599 INFO:tasks.rados.rados.0.smithi079.stdout:256: read oid 25 snap 18 2024-04-24T01:56:11.599 INFO:tasks.rados.rados.0.smithi079.stdout:256: expect (ObjNum 137 snap 17 seq_num 137) 2024-04-24T01:56:11.599 INFO:tasks.rados.rados.0.smithi079.stdout:257: write (excl) oid 30 current snap is 20 2024-04-24T01:56:11.600 INFO:tasks.rados.rados.0.smithi079.stdout:257: seq_num 150 ranges {521050=698057,1877946=643104} 2024-04-24T01:56:11.604 INFO:tasks.rados.rados.0.smithi079.stdout:257: writing smithi07929762-30 from 521050 to 1219107 tid 1 2024-04-24T01:56:11.612 INFO:tasks.rados.rados.0.smithi079.stdout:257: writing smithi07929762-30 from 1877946 to 2521050 tid 2 2024-04-24T01:56:11.612 INFO:tasks.rados.rados.0.smithi079.stdout:258: write oid 3 current snap is 20 2024-04-24T01:56:11.612 INFO:tasks.rados.rados.0.smithi079.stdout:258: seq_num 151 ranges {537857=773306,1936428=520715} 2024-04-24T01:56:11.619 INFO:tasks.rados.rados.0.smithi079.stdout:258: writing smithi07929762-3 from 537857 to 1311163 tid 1 2024-04-24T01:56:11.626 INFO:tasks.rados.rados.0.smithi079.stdout:258: writing smithi07929762-3 from 1936428 to 2457143 tid 2 2024-04-24T01:56:11.626 INFO:tasks.rados.rados.0.smithi079.stdout:259: snap_create 2024-04-24T01:56:11.641 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-04-24T01:56:11.641 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-04-24T01:56:11.641 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T01:56:11.642 INFO:tasks.rados.rados.0.smithi079.stdout:257: finishing write tid 1 to smithi07929762-30 2024-04-24T01:56:11.674 INFO:tasks.rados.rados.0.smithi079.stdout:256: expect (ObjNum 137 snap 17 seq_num 137) 2024-04-24T01:56:11.767 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:11.762+0000 7f165bcad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:11.771 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:11.770+0000 7f165bcad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:11.821 INFO:tasks.rados.rados.0.smithi079.stdout:257: finishing write tid 2 to smithi07929762-30 2024-04-24T01:56:11.821 INFO:tasks.rados.rados.0.smithi079.stdout:257: finishing write tid 3 to smithi07929762-30 2024-04-24T01:56:11.821 INFO:tasks.rados.rados.0.smithi079.stdout:257: finishing write tid 4 to smithi07929762-30 2024-04-24T01:56:11.821 INFO:tasks.rados.rados.0.smithi079.stdout:257: oid 30 updating version 0 to 22 2024-04-24T01:56:11.821 INFO:tasks.rados.rados.0.smithi079.stdout:257: oid 30 updating version 22 to 24 2024-04-24T01:56:11.822 INFO:tasks.rados.rados.0.smithi079.stdout:257: oid 30 version 24 is already newer than 24 2024-04-24T01:56:11.822 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 30 v 24 (ObjNum 150 snap 20 seq_num 150) dirty exists 2024-04-24T01:56:11.822 INFO:tasks.rados.rados.0.smithi079.stdout:257: left oid 30 (ObjNum 150 snap 20 seq_num 150) 2024-04-24T01:56:11.822 INFO:tasks.rados.rados.0.smithi079.stdout:258: finishing write tid 1 to smithi07929762-3 2024-04-24T01:56:11.822 INFO:tasks.rados.rados.0.smithi079.stdout:258: finishing write tid 2 to smithi07929762-3 2024-04-24T01:56:11.822 INFO:tasks.rados.rados.0.smithi079.stdout:258: finishing write tid 3 to smithi07929762-3 2024-04-24T01:56:11.822 INFO:tasks.rados.rados.0.smithi079.stdout:258: finishing write tid 4 to smithi07929762-3 2024-04-24T01:56:11.822 INFO:tasks.rados.rados.0.smithi079.stdout:258: oid 3 updating version 0 to 85 2024-04-24T01:56:11.822 INFO:tasks.rados.rados.0.smithi079.stdout:258: oid 3 version 85 is already newer than 85 2024-04-24T01:56:11.822 INFO:tasks.rados.rados.0.smithi079.stdout:258: oid 3 version 85 is already newer than 85 2024-04-24T01:56:11.822 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 3 v 85 (ObjNum 151 snap 20 seq_num 151) dirty exists 2024-04-24T01:56:11.822 INFO:tasks.rados.rados.0.smithi079.stdout:258: left oid 3 (ObjNum 151 snap 20 seq_num 151) 2024-04-24T01:56:11.958 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:56:11.959 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-24T01:56:11.970 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":77,"stamp":"2024-04-24T01:56:11.369623+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":99986240,"num_objects":40,"num_object_clones":2,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":662,"num_read_kb":331279,"num_write":665,"num_write_kb":144996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":706,"ondisk_log_size":706,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":293408,"kb_used_data":186112,"kb_used_omap":0,"kb_used_meta":107264,"kb_avail":377193952,"statfs":{"total":386547056640,"available":386246606848,"internally_reserved":0,"allocated":190578688,"data_stored":194410370,"data_compressed":89889,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109838336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-6420109,"num_objects":-17,"num_object_clones":-11,"num_object_copies":-51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":-6,"num_read":57,"num_read_kb":51035,"num_write":26,"num_write_kb":5832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.022196"},"pg_stats":[{"pgid":"3.f","version":"19'6","reported_seq":148,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-24T01:56:08.375137+0000","last_change":"2024-04-24T01:56:07.375313+0000","last_active":"2024-04-24T01:56:08.375137+0000","last_peered":"2024-04-24T01:56:08.375137+0000","last_clean":"2024-04-24T01:56:08.375137+0000","last_became_active":"2024-04-24T01:55:30.386726+0000","last_became_peered":"2024-04-24T01:55:30.386726+0000","last_unstale":"2024-04-24T01:56:08.375137+0000","last_undegraded":"2024-04-24T01:56:08.375137+0000","last_fullsized":"2024-04-24T01:56:08.375137+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5310946,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":34,"num_read_kb":28309,"num_write":8,"num_write_kb":2595,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"52'60","reported_seq":190,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-24T01:56:07.167883+0000","last_change":"2024-04-24T01:56:07.226091+0000","last_active":"2024-04-24T01:56:07.167883+0000","last_peered":"2024-04-24T01:56:07.167883+0000","last_clean":"2024-04-24T01:56:07.167883+0000","last_became_active":"2024-04-24T01:55:30.321866+0000","last_became_peered":"2024-04-24T01:55:30.321866+0000","last_unstale":"2024-04-24T01:56:07.167883+0000","last_undegraded":"2024-04-24T01:56:07.167883+0000","last_fullsized":"2024-04-24T01:56:07.167883+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0,"stat_sum":{"num_bytes":6823609,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":47,"num_read_kb":25524,"num_write":48,"num_write_kb":13275,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"53'50","reported_seq":177,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-24T01:56:07.408095+0000","last_change":"2024-04-24T01:56:07.479880+0000","last_active":"2024-04-24T01:56:07.408095+0000","last_peered":"2024-04-24T01:56:07.408095+0000","last_clean":"2024-04-24T01:56:07.408095+0000","last_became_active":"2024-04-24T01:55:30.320186+0000","last_became_peered":"2024-04-24T01:55:30.320186+0000","last_unstale":"2024-04-24T01:56:07.408095+0000","last_undegraded":"2024-04-24T01:56:07.408095+0000","last_fullsized":"2024-04-24T01:56:07.408095+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":5,"snaptrim_duration":0,"stat_sum":{"num_bytes":9361433,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":26,"num_read_kb":8133,"num_write":47,"num_write_kb":12133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"52'58","reported_seq":192,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-24T01:56:08.368342+0000","last_change":"2024-04-24T01:56:07.375079+0000","last_active":"2024-04-24T01:56:08.368342+0000","last_peered":"2024-04-24T01:56:08.368342+0000","last_clean":"2024-04-24T01:56:08.368342+0000","last_became_active":"2024-04-24T01:55:30.288081+0000","last_became_peered":"2024-04-24T01:55:30.288081+0000","last_unstale":"2024-04-24T01:56:08.368342+0000","last_undegraded":"2024-04-24T01:56:08.368342+0000","last_fullsized":"2024-04-24T01:56:08.368342+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":8,"snaptrim_duration":0,"stat_sum":{"num_bytes":8000066,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":52,"num_read_kb":37341,"num_write":37,"num_write_kb":10626,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"39'23","reported_seq":154,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-24T01:56:06.365029+0000","last_change":"2024-04-24T01:56:06.370707+0000","last_active":"2024-04-24T01:56:06.365029+0000","last_peered":"2024-04-24T01:56:06.365029+0000","last_clean":"2024-04-24T01:56:06.365029+0000","last_became_active":"2024-04-24T01:55:30.333815+0000","last_became_peered":"2024-04-24T01:55:30.333815+0000","last_unstale":"2024-04-24T01:56:06.365029+0000","last_undegraded":"2024-04-24T01:56:06.365029+0000","last_fullsized":"2024-04-24T01:56:06.365029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0,"stat_sum":{"num_bytes":3243718,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":22,"num_read_kb":11072,"num_write":25,"num_write_kb":6861,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"54'78","reported_seq":228,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-24T01:56:08.407401+0000","last_change":"2024-04-24T01:56:07.457739+0000","last_active":"2024-04-24T01:56:08.407401+0000","last_peered":"2024-04-24T01:56:08.407401+0000","last_clean":"2024-04-24T01:56:08.407401+0000","last_became_active":"2024-04-24T01:55:30.315075+0000","last_became_peered":"2024-04-24T01:55:30.315075+0000","last_unstale":"2024-04-24T01:56:08.407401+0000","last_undegraded":"2024-04-24T01:56:08.407401+0000","last_fullsized":"2024-04-24T01:56:08.407401+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":10,"snaptrim_duration":0,"stat_sum":{"num_bytes":12033713,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":80,"num_read_kb":47317,"num_write":71,"num_write_kb":19359,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"53'18","reported_seq":145,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-24T01:56:08.377628+0000","last_change":"2024-04-24T01:56:07.443958+0000","last_active":"2024-04-24T01:56:08.377628+0000","last_peered":"2024-04-24T01:56:08.377628+0000","last_clean":"2024-04-24T01:56:08.377628+0000","last_became_active":"2024-04-24T01:55:30.350300+0000","last_became_peered":"2024-04-24T01:55:30.350300+0000","last_unstale":"2024-04-24T01:56:08.377628+0000","last_undegraded":"2024-04-24T01:56:08.377628+0000","last_fullsized":"2024-04-24T01:56:08.377628+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":5003,"num_write":15,"num_write_kb":3526,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"47'5","reported_seq":127,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-24T01:56:07.388589+0000","last_change":"2024-04-24T01:56:07.398280+0000","last_active":"2024-04-24T01:56:07.388589+0000","last_peered":"2024-04-24T01:56:07.388589+0000","last_clean":"2024-04-24T01:56:07.388589+0000","last_became_active":"2024-04-24T01:55:30.376813+0000","last_became_peered":"2024-04-24T01:55:30.376813+0000","last_unstale":"2024-04-24T01:56:07.388589+0000","last_undegraded":"2024-04-24T01:56:07.388589+0000","last_fullsized":"2024-04-24T01:56:07.388589+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'97","reported_seq":248,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-24T01:56:06.355514+0000","last_change":"2024-04-24T01:55:16.196470+0000","last_active":"2024-04-24T01:56:06.355514+0000","last_peered":"2024-04-24T01:56:06.355514+0000","last_clean":"2024-04-24T01:56:06.355514+0000","last_became_active":"2024-04-24T01:55:16.190464+0000","last_became_peered":"2024-04-24T01:55:16.190464+0000","last_unstale":"2024-04-24T01:56:06.355514+0000","last_undegraded":"2024-04-24T01:56:06.355514+0000","last_fullsized":"2024-04-24T01:56:06.355514+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:15.127115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:15.127115+0000","last_clean_scrub_stamp":"2024-04-24T01:55:15.127115+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":114,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-24T01:56:07.386979+0000","last_change":"2024-04-24T01:55:18.236148+0000","last_active":"2024-04-24T01:56:07.386979+0000","last_peered":"2024-04-24T01:56:07.386979+0000","last_clean":"2024-04-24T01:56:07.386979+0000","last_became_active":"2024-04-24T01:55:18.230154+0000","last_became_peered":"2024-04-24T01:55:18.230154+0000","last_unstale":"2024-04-24T01:56:07.386979+0000","last_undegraded":"2024-04-24T01:56:07.386979+0000","last_fullsized":"2024-04-24T01:56:07.386979+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"53'64","reported_seq":209,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-24T01:56:07.444949+0000","last_change":"2024-04-24T01:56:07.461776+0000","last_active":"2024-04-24T01:56:07.444949+0000","last_peered":"2024-04-24T01:56:07.444949+0000","last_clean":"2024-04-24T01:56:07.444949+0000","last_became_active":"2024-04-24T01:55:30.316222+0000","last_became_peered":"2024-04-24T01:55:30.316222+0000","last_unstale":"2024-04-24T01:56:07.444949+0000","last_undegraded":"2024-04-24T01:56:07.444949+0000","last_fullsized":"2024-04-24T01:56:07.444949+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0,"stat_sum":{"num_bytes":6504309,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":68,"num_read_kb":41075,"num_write":55,"num_write_kb":15303,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":117,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-24T01:56:08.380596+0000","last_change":"2024-04-24T01:55:18.265905+0000","last_active":"2024-04-24T01:56:08.380596+0000","last_peered":"2024-04-24T01:56:08.380596+0000","last_clean":"2024-04-24T01:56:08.380596+0000","last_became_active":"2024-04-24T01:55:18.257020+0000","last_became_peered":"2024-04-24T01:55:18.257020+0000","last_unstale":"2024-04-24T01:56:08.380596+0000","last_undegraded":"2024-04-24T01:56:08.380596+0000","last_fullsized":"2024-04-24T01:56:08.380596+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"52'34","reported_seq":170,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-24T01:56:06.367424+0000","last_change":"2024-04-24T01:56:06.411410+0000","last_active":"2024-04-24T01:56:06.367424+0000","last_peered":"2024-04-24T01:56:06.367424+0000","last_clean":"2024-04-24T01:56:06.367424+0000","last_became_active":"2024-04-24T01:55:30.344129+0000","last_became_peered":"2024-04-24T01:55:30.344129+0000","last_unstale":"2024-04-24T01:56:06.367424+0000","last_undegraded":"2024-04-24T01:56:06.367424+0000","last_fullsized":"2024-04-24T01:56:06.367424+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0,"stat_sum":{"num_bytes":9109328,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":38,"num_read_kb":19958,"num_write":32,"num_write_kb":8371,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":114,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-24T01:56:07.377447+0000","last_change":"2024-04-24T01:55:18.215251+0000","last_active":"2024-04-24T01:56:07.377447+0000","last_peered":"2024-04-24T01:56:07.377447+0000","last_clean":"2024-04-24T01:56:07.377447+0000","last_became_active":"2024-04-24T01:55:18.206605+0000","last_became_peered":"2024-04-24T01:55:18.206605+0000","last_unstale":"2024-04-24T01:56:07.377447+0000","last_undegraded":"2024-04-24T01:56:07.377447+0000","last_fullsized":"2024-04-24T01:56:07.377447+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"39'22","reported_seq":149,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-24T01:56:06.367061+0000","last_change":"2024-04-24T01:56:06.370509+0000","last_active":"2024-04-24T01:56:06.367061+0000","last_peered":"2024-04-24T01:56:06.367061+0000","last_clean":"2024-04-24T01:56:06.367061+0000","last_became_active":"2024-04-24T01:55:30.401366+0000","last_became_peered":"2024-04-24T01:55:30.401366+0000","last_unstale":"2024-04-24T01:56:06.367061+0000","last_undegraded":"2024-04-24T01:56:06.367061+0000","last_fullsized":"2024-04-24T01:56:06.367061+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0,"stat_sum":{"num_bytes":7042067,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":9364,"num_write":18,"num_write_kb":5026,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":117,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-24T01:56:08.381431+0000","last_change":"2024-04-24T01:55:18.230893+0000","last_active":"2024-04-24T01:56:08.381431+0000","last_peered":"2024-04-24T01:56:08.381431+0000","last_clean":"2024-04-24T01:56:08.381431+0000","last_became_active":"2024-04-24T01:55:18.225188+0000","last_became_peered":"2024-04-24T01:55:18.225188+0000","last_unstale":"2024-04-24T01:56:08.381431+0000","last_undegraded":"2024-04-24T01:56:08.381431+0000","last_fullsized":"2024-04-24T01:56:08.381431+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"37'41","reported_seq":168,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-24T01:56:07.149447+0000","last_change":"2024-04-24T01:56:07.158588+0000","last_active":"2024-04-24T01:56:07.149447+0000","last_peered":"2024-04-24T01:56:07.149447+0000","last_clean":"2024-04-24T01:56:07.149447+0000","last_became_active":"2024-04-24T01:55:30.343318+0000","last_became_peered":"2024-04-24T01:55:30.343318+0000","last_unstale":"2024-04-24T01:56:07.149447+0000","last_undegraded":"2024-04-24T01:56:07.149447+0000","last_fullsized":"2024-04-24T01:56:07.149447+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":3,"snaptrim_duration":0,"stat_sum":{"num_bytes":10722767,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":32,"num_read_kb":20292,"num_write":36,"num_write_kb":11038,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"52'52","reported_seq":189,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-24T01:56:07.392063+0000","last_change":"2024-04-24T01:56:07.399043+0000","last_active":"2024-04-24T01:56:07.392063+0000","last_peered":"2024-04-24T01:56:07.392063+0000","last_clean":"2024-04-24T01:56:07.392063+0000","last_became_active":"2024-04-24T01:55:30.345308+0000","last_became_peered":"2024-04-24T01:55:30.345308+0000","last_unstale":"2024-04-24T01:56:07.392063+0000","last_undegraded":"2024-04-24T01:56:07.392063+0000","last_fullsized":"2024-04-24T01:56:07.392063+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0,"stat_sum":{"num_bytes":10184976,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":52,"num_read_kb":30125,"num_write":41,"num_write_kb":10356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":111,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-24T01:56:07.159433+0000","last_change":"2024-04-24T01:55:18.224238+0000","last_active":"2024-04-24T01:56:07.159433+0000","last_peered":"2024-04-24T01:56:07.159433+0000","last_clean":"2024-04-24T01:56:07.159433+0000","last_became_active":"2024-04-24T01:55:18.219131+0000","last_became_peered":"2024-04-24T01:55:18.219131+0000","last_unstale":"2024-04-24T01:56:07.159433+0000","last_undegraded":"2024-04-24T01:56:07.159433+0000","last_fullsized":"2024-04-24T01:56:07.159433+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"52'21","reported_seq":148,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-24T01:56:07.165334+0000","last_change":"2024-04-24T01:56:07.213312+0000","last_active":"2024-04-24T01:56:07.165334+0000","last_peered":"2024-04-24T01:56:07.165334+0000","last_clean":"2024-04-24T01:56:07.165334+0000","last_became_active":"2024-04-24T01:55:30.319958+0000","last_became_peered":"2024-04-24T01:55:30.319958+0000","last_unstale":"2024-04-24T01:56:07.165334+0000","last_undegraded":"2024-04-24T01:56:07.165334+0000","last_fullsized":"2024-04-24T01:56:07.165334+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336173,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":19605,"num_write":23,"num_write_kb":6671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":115,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-24T01:56:06.368472+0000","last_change":"2024-04-24T01:55:18.235763+0000","last_active":"2024-04-24T01:56:06.368472+0000","last_peered":"2024-04-24T01:56:06.368472+0000","last_clean":"2024-04-24T01:56:06.368472+0000","last_became_active":"2024-04-24T01:55:18.229175+0000","last_became_peered":"2024-04-24T01:55:18.229175+0000","last_unstale":"2024-04-24T01:56:06.368472+0000","last_undegraded":"2024-04-24T01:56:06.368472+0000","last_fullsized":"2024-04-24T01:56:06.368472+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"39'19","reported_seq":151,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-24T01:56:06.358084+0000","last_change":"2024-04-24T01:56:06.364686+0000","last_active":"2024-04-24T01:56:06.358084+0000","last_peered":"2024-04-24T01:56:06.358084+0000","last_clean":"2024-04-24T01:56:06.358084+0000","last_became_active":"2024-04-24T01:55:30.372690+0000","last_became_peered":"2024-04-24T01:55:30.372690+0000","last_unstale":"2024-04-24T01:56:06.358084+0000","last_undegraded":"2024-04-24T01:56:06.358084+0000","last_fullsized":"2024-04-24T01:56:06.358084+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0,"stat_sum":{"num_bytes":3462209,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":26,"num_read_kb":16700,"num_write":19,"num_write_kb":5298,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":111,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-24T01:56:07.158216+0000","last_change":"2024-04-24T01:55:18.278351+0000","last_active":"2024-04-24T01:56:07.158216+0000","last_peered":"2024-04-24T01:56:07.158216+0000","last_clean":"2024-04-24T01:56:07.158216+0000","last_became_active":"2024-04-24T01:55:18.273668+0000","last_became_peered":"2024-04-24T01:55:18.273668+0000","last_unstale":"2024-04-24T01:56:07.158216+0000","last_undegraded":"2024-04-24T01:56:07.158216+0000","last_fullsized":"2024-04-24T01:56:07.158216+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"52'58","reported_seq":180,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-24T01:56:07.238659+0000","last_change":"2024-04-24T01:56:07.258617+0000","last_active":"2024-04-24T01:56:07.238659+0000","last_peered":"2024-04-24T01:56:07.238659+0000","last_clean":"2024-04-24T01:56:07.238659+0000","last_became_active":"2024-04-24T01:55:30.363123+0000","last_became_peered":"2024-04-24T01:55:30.363123+0000","last_unstale":"2024-04-24T01:56:07.238659+0000","last_undegraded":"2024-04-24T01:56:07.238659+0000","last_fullsized":"2024-04-24T01:56:07.238659+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0,"stat_sum":{"num_bytes":5260558,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":31,"num_read_kb":11369,"num_write":48,"num_write_kb":11816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":114,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-24T01:56:07.389505+0000","last_change":"2024-04-24T01:55:18.241117+0000","last_active":"2024-04-24T01:56:07.389505+0000","last_peered":"2024-04-24T01:56:07.389505+0000","last_clean":"2024-04-24T01:56:07.389505+0000","last_became_active":"2024-04-24T01:55:18.234313+0000","last_became_peered":"2024-04-24T01:55:18.234313+0000","last_unstale":"2024-04-24T01:56:07.389505+0000","last_undegraded":"2024-04-24T01:56:07.389505+0000","last_fullsized":"2024-04-24T01:56:07.389505+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":99395872,"num_objects":38,"num_object_clones":2,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":559,"num_read_kb":331188,"num_write":528,"num_write_kb":143102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":173813760,"data_stored":174926858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":609,"ondisk_log_size":609,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":1}],"osd_stats":[{"osd":3,"up_from":10,"seq":42949673014,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":77844,"kb_used_data":51020,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94293996,"statfs":{"total":96636764160,"available":96557051904,"internally_reserved":0,"allocated":52244480,"data_stored":52935406,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673014,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":64804,"kb_used_data":37980,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94307036,"statfs":{"total":96636764160,"available":96570404864,"internally_reserved":0,"allocated":38891520,"data_stored":39694537,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705719,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":67152,"kb_used_data":40328,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94304688,"statfs":{"total":96636764160,"available":96568000512,"internally_reserved":0,"allocated":41295872,"data_stored":42100798,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705718,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":83608,"kb_used_data":56784,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94288232,"statfs":{"total":96636764160,"available":96551149568,"internally_reserved":0,"allocated":58146816,"data_stored":59679629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":41766912,"data_stored":42300745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40923136,"data_stored":40549954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":39370752,"data_stored":39898145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":51752960,"data_stored":52178014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T01:56:12.377 INFO:tasks.rados.rados.0.smithi079.stdout:256: done (3 left) 2024-04-24T01:56:12.377 INFO:tasks.rados.rados.0.smithi079.stdout:257: done (2 left) 2024-04-24T01:56:12.378 INFO:tasks.rados.rados.0.smithi079.stdout:258: done (1 left) 2024-04-24T01:56:12.378 INFO:tasks.rados.rados.0.smithi079.stdout:259: done (0 left) 2024-04-24T01:56:12.378 INFO:tasks.rados.rados.0.smithi079.stdout:260: write (excl) oid 45 current snap is 21 2024-04-24T01:56:12.378 INFO:tasks.rados.rados.0.smithi079.stdout:260: seq_num 152 ranges {554664=448555,1478557=664373} 2024-04-24T01:56:12.382 INFO:tasks.rados.rados.0.smithi079.stdout:260: writing smithi07929762-45 from 554664 to 1003219 tid 1 2024-04-24T01:56:12.389 INFO:tasks.rados.rados.0.smithi079.stdout:260: writing smithi07929762-45 from 1478557 to 2142930 tid 2 2024-04-24T01:56:12.389 INFO:tasks.rados.rados.0.smithi079.stdout:261: snap_remove snap 17 2024-04-24T01:56:12.974 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T01:56:13.082 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:13.081+0000 7f3126864640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:13.086 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:13.086+0000 7f3126864640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:13.386 INFO:tasks.rados.rados.0.smithi079.stdout:261: done (1 left) 2024-04-24T01:56:13.386 INFO:tasks.rados.rados.0.smithi079.stdout:262: write (excl) oid 9 current snap is 21 2024-04-24T01:56:13.387 INFO:tasks.rados.rados.0.smithi079.stdout:260: finishing write tid 1 to smithi07929762-45 2024-04-24T01:56:13.387 INFO:tasks.rados.rados.0.smithi079.stdout:262: seq_num 153 ranges {571471=640157,1653392=524384} 2024-04-24T01:56:13.393 INFO:tasks.rados.rados.0.smithi079.stdout:262: writing smithi07929762-9 from 571471 to 1211628 tid 1 2024-04-24T01:56:13.398 INFO:tasks.rados.rados.0.smithi079.stdout:262: writing smithi07929762-9 from 1653392 to 2177776 tid 2 2024-04-24T01:56:13.398 INFO:tasks.rados.rados.0.smithi079.stdout:263: write (excl) oid 20 current snap is 21 2024-04-24T01:56:13.398 INFO:tasks.rados.rados.0.smithi079.stdout:263: seq_num 154 ranges {588278=715406,1711874=668042} 2024-04-24T01:56:13.405 INFO:tasks.rados.rados.0.smithi079.stdout:263: writing smithi07929762-20 from 588278 to 1303684 tid 1 2024-04-24T01:56:13.412 INFO:tasks.rados.rados.0.smithi079.stdout:263: writing smithi07929762-20 from 1711874 to 2379916 tid 2 2024-04-24T01:56:13.412 INFO:tasks.rados.rados.0.smithi079.stdout:264: snap_create 2024-04-24T01:56:13.412 INFO:tasks.rados.rados.0.smithi079.stdout:260: finishing write tid 2 to smithi07929762-45 2024-04-24T01:56:13.412 INFO:tasks.rados.rados.0.smithi079.stdout:260: finishing write tid 3 to smithi07929762-45 2024-04-24T01:56:13.412 INFO:tasks.rados.rados.0.smithi079.stdout:260: finishing write tid 4 to smithi07929762-45 2024-04-24T01:56:13.412 INFO:tasks.rados.rados.0.smithi079.stdout:260: oid 45 updating version 0 to 69 2024-04-24T01:56:13.412 INFO:tasks.rados.rados.0.smithi079.stdout:260: oid 45 version 69 is already newer than 69 2024-04-24T01:56:13.413 INFO:tasks.rados.rados.0.smithi079.stdout:260: oid 45 version 69 is already newer than 67 2024-04-24T01:56:13.413 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 45 v 69 (ObjNum 152 snap 21 seq_num 152) dirty exists 2024-04-24T01:56:13.413 INFO:tasks.rados.rados.0.smithi079.stdout:260: left oid 45 (ObjNum 152 snap 21 seq_num 152) 2024-04-24T01:56:13.460 INFO:tasks.rados.rados.0.smithi079.stdout:262: finishing write tid 1 to smithi07929762-9 2024-04-24T01:56:13.477 INFO:tasks.rados.rados.0.smithi079.stdout:262: finishing write tid 2 to smithi07929762-9 2024-04-24T01:56:13.480 INFO:tasks.rados.rados.0.smithi079.stdout:262: finishing write tid 3 to smithi07929762-9 2024-04-24T01:56:13.484 INFO:tasks.rados.rados.0.smithi079.stdout:262: finishing write tid 4 to smithi07929762-9 2024-04-24T01:56:13.484 INFO:tasks.rados.rados.0.smithi079.stdout:262: oid 9 updating version 0 to 56 2024-04-24T01:56:13.484 INFO:tasks.rados.rados.0.smithi079.stdout:262: oid 9 version 56 is already newer than 54 2024-04-24T01:56:13.484 INFO:tasks.rados.rados.0.smithi079.stdout:262: oid 9 version 56 is already newer than 56 2024-04-24T01:56:13.484 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 9 v 56 (ObjNum 153 snap 21 seq_num 153) dirty exists 2024-04-24T01:56:13.484 INFO:tasks.rados.rados.0.smithi079.stdout:262: left oid 9 (ObjNum 153 snap 21 seq_num 153) 2024-04-24T01:56:13.512 INFO:tasks.rados.rados.0.smithi079.stdout:263: finishing write tid 1 to smithi07929762-20 2024-04-24T01:56:13.530 INFO:tasks.rados.rados.0.smithi079.stdout:263: finishing write tid 2 to smithi07929762-20 2024-04-24T01:56:13.537 INFO:tasks.rados.rados.0.smithi079.stdout:263: finishing write tid 3 to smithi07929762-20 2024-04-24T01:56:13.538 INFO:tasks.rados.rados.0.smithi079.stdout:263: finishing write tid 4 to smithi07929762-20 2024-04-24T01:56:13.538 INFO:tasks.rados.rados.0.smithi079.stdout:263: oid 20 updating version 0 to 60 2024-04-24T01:56:13.538 INFO:tasks.rados.rados.0.smithi079.stdout:263: oid 20 version 60 is already newer than 60 2024-04-24T01:56:13.538 INFO:tasks.rados.rados.0.smithi079.stdout:263: oid 20 version 60 is already newer than 58 2024-04-24T01:56:13.538 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 20 v 60 (ObjNum 154 snap 21 seq_num 154) dirty exists 2024-04-24T01:56:13.538 INFO:tasks.rados.rados.0.smithi079.stdout:263: left oid 20 (ObjNum 154 snap 21 seq_num 154) 2024-04-24T01:56:14.382 INFO:tasks.rados.rados.0.smithi079.stdout:260: done (3 left) 2024-04-24T01:56:14.382 INFO:tasks.rados.rados.0.smithi079.stdout:262: done (2 left) 2024-04-24T01:56:14.382 INFO:tasks.rados.rados.0.smithi079.stdout:263: done (1 left) 2024-04-24T01:56:14.382 INFO:tasks.rados.rados.0.smithi079.stdout:264: done (0 left) 2024-04-24T01:56:14.383 INFO:tasks.rados.rados.0.smithi079.stdout:265: snap_create 2024-04-24T01:56:15.193 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T01:56:15.310 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:15.309+0000 7ff1a4eed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:15.314 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:15.314+0000 7ff1a4eed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:15.387 INFO:tasks.rados.rados.0.smithi079.stdout:265: done (0 left) 2024-04-24T01:56:15.387 INFO:tasks.rados.rados.0.smithi079.stdout:266: snap_remove snap 22 2024-04-24T01:56:16.390 INFO:tasks.rados.rados.0.smithi079.stdout:266: done (0 left) 2024-04-24T01:56:16.390 INFO:tasks.rados.rados.0.smithi079.stdout:267: write (excl) oid 9 current snap is 23 2024-04-24T01:56:16.390 INFO:tasks.rados.rados.0.smithi079.stdout:267: seq_num 155 ranges {605085=790655,2170356=434729} 2024-04-24T01:56:16.398 INFO:tasks.rados.rados.0.smithi079.stdout:267: writing smithi07929762-9 from 605085 to 1395740 tid 1 2024-04-24T01:56:16.405 INFO:tasks.rados.rados.0.smithi079.stdout:267: writing smithi07929762-9 from 2170356 to 2605085 tid 2 2024-04-24T01:56:16.405 INFO:tasks.rados.rados.0.smithi079.stdout:268: snap_create 2024-04-24T01:56:16.503 INFO:tasks.rados.rados.0.smithi079.stdout:267: finishing write tid 1 to smithi07929762-9 2024-04-24T01:56:16.522 INFO:tasks.rados.rados.0.smithi079.stdout:267: finishing write tid 2 to smithi07929762-9 2024-04-24T01:56:16.523 INFO:tasks.rados.rados.0.smithi079.stdout:267: finishing write tid 3 to smithi07929762-9 2024-04-24T01:56:16.524 INFO:tasks.rados.rados.0.smithi079.stdout:267: finishing write tid 4 to smithi07929762-9 2024-04-24T01:56:16.524 INFO:tasks.rados.rados.0.smithi079.stdout:267: oid 9 updating version 0 to 64 2024-04-24T01:56:16.524 INFO:tasks.rados.rados.0.smithi079.stdout:267: oid 9 version 64 is already newer than 64 2024-04-24T01:56:16.524 INFO:tasks.rados.rados.0.smithi079.stdout:267: oid 9 version 64 is already newer than 62 2024-04-24T01:56:16.524 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 9 v 64 (ObjNum 155 snap 23 seq_num 155) dirty exists 2024-04-24T01:56:16.525 INFO:tasks.rados.rados.0.smithi079.stdout:267: left oid 9 (ObjNum 155 snap 23 seq_num 155) 2024-04-24T01:56:16.973 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-04-24T01:56:16.973 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-04-24T01:56:16.973 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-04-24T01:56:16.973 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-04-24T01:56:17.085 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:17.078+0000 7fb0018ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:17.086 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:17.086+0000 7fb0018ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:17.101 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T01:56:17.101 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T01:56:17.101 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T01:56:17.101 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T01:56:17.101 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T01:56:17.101 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T01:56:17.101 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T01:56:17.101 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T01:56:17.101 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T01:56:17.101 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T01:56:17.101 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T01:56:17.102 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T01:56:17.104 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T01:56:17.104 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T01:56:17.104 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-04-24T01:56:17.211 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:17.207+0000 7fdf0ce2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:17.215 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:17.215+0000 7fdf0ce2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:17.229 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T01:56:17.229 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T01:56:17.230 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T01:56:17.230 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T01:56:17.230 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T01:56:17.230 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T01:56:17.230 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T01:56:17.230 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T01:56:17.230 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T01:56:17.230 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T01:56:17.230 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T01:56:17.230 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T01:56:17.232 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T01:56:17.233 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T01:56:17.233 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-04-24T01:56:17.339 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:17.334+0000 7f24b9e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:17.343 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:17.343+0000 7f24b9e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:17.360 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T01:56:17.360 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T01:56:17.360 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T01:56:17.361 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T01:56:17.361 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T01:56:17.361 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T01:56:17.361 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T01:56:17.361 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T01:56:17.361 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T01:56:17.361 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T01:56:17.361 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T01:56:17.361 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T01:56:17.363 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T01:56:17.364 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T01:56:17.364 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-04-24T01:56:17.400 INFO:tasks.rados.rados.0.smithi079.stdout:267: done (1 left) 2024-04-24T01:56:17.400 INFO:tasks.rados.rados.0.smithi079.stdout:268: done (0 left) 2024-04-24T01:56:17.400 INFO:tasks.rados.rados.0.smithi079.stdout:269: read oid 33 snap -1 2024-04-24T01:56:17.400 INFO:tasks.rados.rados.0.smithi079.stdout:269: expect (ObjNum 124 snap 15 seq_num 124) 2024-04-24T01:56:17.400 INFO:tasks.rados.rados.0.smithi079.stdout:270: delete oid 39 current snap is 24 2024-04-24T01:56:17.416 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T01:56:17.453 INFO:tasks.rados.rados.0.smithi079.stdout:269: expect (ObjNum 124 snap 15 seq_num 124) 2024-04-24T01:56:17.493 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:17.486+0000 7fd66726d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:17.495 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:17.496+0000 7fd66726d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:17.511 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T01:56:17.511 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T01:56:17.512 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T01:56:17.512 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T01:56:17.512 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T01:56:17.512 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T01:56:17.512 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T01:56:17.512 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T01:56:17.512 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T01:56:17.512 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T01:56:17.512 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T01:56:17.512 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T01:56:17.514 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T01:56:17.515 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T01:56:17.540 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:17.539+0000 7f8807f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:17.544 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:17.544+0000 7f8807f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:17.567 INFO:tasks.rados.rados.0.smithi079.stdout:269: done (1 left) 2024-04-24T01:56:17.567 INFO:tasks.rados.rados.0.smithi079.stdout:270: done (0 left) 2024-04-24T01:56:17.568 INFO:tasks.rados.rados.0.smithi079.stdout:271: snap_remove snap 21 2024-04-24T01:56:18.397 INFO:tasks.rados.rados.0.smithi079.stdout:271: done (0 left) 2024-04-24T01:56:18.397 INFO:tasks.rados.rados.0.smithi079.stdout:272: write (excl) oid 2 current snap is 24 2024-04-24T01:56:18.398 INFO:tasks.rados.rados.0.smithi079.stdout:272: seq_num 157 ranges {638699=541153,1770967=531722} 2024-04-24T01:56:18.403 INFO:tasks.rados.rados.0.smithi079.stdout:272: writing smithi07929762-2 from 638699 to 1179852 tid 1 2024-04-24T01:56:18.410 INFO:tasks.rados.rados.0.smithi079.stdout:272: writing smithi07929762-2 from 1770967 to 2302689 tid 2 2024-04-24T01:56:18.410 INFO:tasks.rados.rados.0.smithi079.stdout:273: delete oid 8 current snap is 24 2024-04-24T01:56:18.445 INFO:tasks.rados.rados.0.smithi079.stdout:273: done (1 left) 2024-04-24T01:56:18.445 INFO:tasks.rados.rados.0.smithi079.stdout:274: delete oid 42 current snap is 24 2024-04-24T01:56:18.490 INFO:tasks.rados.rados.0.smithi079.stdout:274: done (1 left) 2024-04-24T01:56:18.490 INFO:tasks.rados.rados.0.smithi079.stdout:275: read oid 27 snap -1 2024-04-24T01:56:18.490 INFO:tasks.rados.rados.0.smithi079.stdout:275: expect (ObjNum 74 snap 3 seq_num 74) 2024-04-24T01:56:18.490 INFO:tasks.rados.rados.0.smithi079.stdout:276: snap_remove snap 20 2024-04-24T01:56:19.403 INFO:tasks.rados.rados.0.smithi079.stdout:276: done (2 left) 2024-04-24T01:56:19.404 INFO:tasks.rados.rados.0.smithi079.stdout:277: delete oid 32 current snap is 24 2024-04-24T01:56:19.404 INFO:tasks.rados.rados.0.smithi079.stdout:272: finishing write tid 1 to smithi07929762-2 2024-04-24T01:56:19.404 INFO:tasks.rados.rados.0.smithi079.stdout:272: finishing write tid 2 to smithi07929762-2 2024-04-24T01:56:19.404 INFO:tasks.rados.rados.0.smithi079.stdout:272: finishing write tid 3 to smithi07929762-2 2024-04-24T01:56:19.404 INFO:tasks.rados.rados.0.smithi079.stdout:272: finishing write tid 4 to smithi07929762-2 2024-04-24T01:56:19.404 INFO:tasks.rados.rados.0.smithi079.stdout:272: oid 2 updating version 0 to 27 2024-04-24T01:56:19.404 INFO:tasks.rados.rados.0.smithi079.stdout:272: oid 2 version 27 is already newer than 27 2024-04-24T01:56:19.404 INFO:tasks.rados.rados.0.smithi079.stdout:272: oid 2 version 27 is already newer than 25 2024-04-24T01:56:19.404 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 2 v 27 (ObjNum 157 snap 24 seq_num 157) dirty exists 2024-04-24T01:56:19.404 INFO:tasks.rados.rados.0.smithi079.stdout:272: left oid 2 (ObjNum 157 snap 24 seq_num 157) 2024-04-24T01:56:19.404 INFO:tasks.rados.rados.0.smithi079.stdout:275: expect (ObjNum 74 snap 3 seq_num 74) 2024-04-24T01:56:19.531 INFO:tasks.rados.rados.0.smithi079.stdout:272: done (2 left) 2024-04-24T01:56:19.531 INFO:tasks.rados.rados.0.smithi079.stdout:275: done (1 left) 2024-04-24T01:56:19.531 INFO:tasks.rados.rados.0.smithi079.stdout:277: done (0 left) 2024-04-24T01:56:19.531 INFO:tasks.rados.rados.0.smithi079.stdout:278: write (excl) oid 37 current snap is 24 2024-04-24T01:56:19.531 INFO:tasks.rados.rados.0.smithi079.stdout:278: seq_num 161 ranges {705927=558502,2004895=422707} 2024-04-24T01:56:19.538 INFO:tasks.rados.rados.0.smithi079.stdout:278: writing smithi07929762-37 from 705927 to 1264429 tid 1 2024-04-24T01:56:19.544 INFO:tasks.rados.rados.0.smithi079.stdout:278: writing smithi07929762-37 from 2004895 to 2427602 tid 2 2024-04-24T01:56:19.544 INFO:tasks.rados.rados.0.smithi079.stdout:279: write (excl) oid 50 current snap is 24 2024-04-24T01:56:19.545 INFO:tasks.rados.rados.0.smithi079.stdout:279: seq_num 162 ranges {722734=633751,2063377=659357} 2024-04-24T01:56:19.552 INFO:tasks.rados.rados.0.smithi079.stdout:279: writing smithi07929762-50 from 722734 to 1356485 tid 1 2024-04-24T01:56:19.564 INFO:tasks.rados.rados.0.smithi079.stdout:279: writing smithi07929762-50 from 2063377 to 2722734 tid 2 2024-04-24T01:56:19.564 INFO:tasks.rados.rados.0.smithi079.stdout:280: snap_remove snap 18 2024-04-24T01:56:19.638 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T01:56:19.741 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:19.740+0000 7f2c95a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:19.746 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:19.745+0000 7f2c95a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:20.404 INFO:tasks.rados.rados.0.smithi079.stdout:280: done (2 left) 2024-04-24T01:56:20.404 INFO:tasks.rados.rados.0.smithi079.stdout:281: write (excl) oid 6 current snap is 24 2024-04-24T01:56:20.404 INFO:tasks.rados.rados.0.smithi079.stdout:281: seq_num 163 ranges {739541=709000,2121859=426376} 2024-04-24T01:56:20.412 INFO:tasks.rados.rados.0.smithi079.stdout:281: writing smithi07929762-6 from 739541 to 1448541 tid 1 2024-04-24T01:56:20.418 INFO:tasks.rados.rados.0.smithi079.stdout:281: writing smithi07929762-6 from 2121859 to 2548235 tid 2 2024-04-24T01:56:20.419 INFO:tasks.rados.rados.0.smithi079.stdout:282: write oid 7 current snap is 24 2024-04-24T01:56:20.419 INFO:tasks.rados.rados.0.smithi079.stdout:282: seq_num 164 ranges {756348=784249,2063988=686387} 2024-04-24T01:56:20.426 INFO:tasks.rados.rados.0.smithi079.stdout:282: writing smithi07929762-7 from 756348 to 1540597 tid 1 2024-04-24T01:56:20.433 INFO:tasks.rados.rados.0.smithi079.stdout:282: writing smithi07929762-7 from 2063988 to 2750375 tid 2 2024-04-24T01:56:20.433 INFO:tasks.rados.rados.0.smithi079.stdout:283: snap_create 2024-04-24T01:56:20.434 INFO:tasks.rados.rados.0.smithi079.stdout:278: finishing write tid 1 to smithi07929762-37 2024-04-24T01:56:20.434 INFO:tasks.rados.rados.0.smithi079.stdout:279: finishing write tid 1 to smithi07929762-50 2024-04-24T01:56:20.434 INFO:tasks.rados.rados.0.smithi079.stdout:278: finishing write tid 2 to smithi07929762-37 2024-04-24T01:56:20.434 INFO:tasks.rados.rados.0.smithi079.stdout:278: finishing write tid 3 to smithi07929762-37 2024-04-24T01:56:20.434 INFO:tasks.rados.rados.0.smithi079.stdout:278: finishing write tid 4 to smithi07929762-37 2024-04-24T01:56:20.434 INFO:tasks.rados.rados.0.smithi079.stdout:278: oid 37 updating version 0 to 89 2024-04-24T01:56:20.434 INFO:tasks.rados.rados.0.smithi079.stdout:278: oid 37 updating version 89 to 91 2024-04-24T01:56:20.434 INFO:tasks.rados.rados.0.smithi079.stdout:278: oid 37 version 91 is already newer than 91 2024-04-24T01:56:20.434 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 37 v 91 (ObjNum 161 snap 24 seq_num 161) dirty exists 2024-04-24T01:56:20.435 INFO:tasks.rados.rados.0.smithi079.stdout:278: left oid 37 (ObjNum 161 snap 24 seq_num 161) 2024-04-24T01:56:20.435 INFO:tasks.rados.rados.0.smithi079.stdout:279: finishing write tid 2 to smithi07929762-50 2024-04-24T01:56:20.435 INFO:tasks.rados.rados.0.smithi079.stdout:279: finishing write tid 3 to smithi07929762-50 2024-04-24T01:56:20.435 INFO:tasks.rados.rados.0.smithi079.stdout:279: finishing write tid 4 to smithi07929762-50 2024-04-24T01:56:20.435 INFO:tasks.rados.rados.0.smithi079.stdout:279: oid 50 updating version 0 to 63 2024-04-24T01:56:20.435 INFO:tasks.rados.rados.0.smithi079.stdout:279: oid 50 version 63 is already newer than 61 2024-04-24T01:56:20.435 INFO:tasks.rados.rados.0.smithi079.stdout:279: oid 50 version 63 is already newer than 63 2024-04-24T01:56:20.435 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 50 v 63 (ObjNum 162 snap 24 seq_num 162) dirty exists 2024-04-24T01:56:20.435 INFO:tasks.rados.rados.0.smithi079.stdout:279: left oid 50 (ObjNum 162 snap 24 seq_num 162) 2024-04-24T01:56:20.511 INFO:tasks.rados.rados.0.smithi079.stdout:282: finishing write tid 1 to smithi07929762-7 2024-04-24T01:56:20.519 INFO:tasks.rados.rados.0.smithi079.stdout:281: finishing write tid 1 to smithi07929762-6 2024-04-24T01:56:20.527 INFO:tasks.rados.rados.0.smithi079.stdout:282: finishing write tid 2 to smithi07929762-7 2024-04-24T01:56:20.532 INFO:tasks.rados.rados.0.smithi079.stdout:282: finishing write tid 3 to smithi07929762-7 2024-04-24T01:56:20.534 INFO:tasks.rados.rados.0.smithi079.stdout:282: finishing write tid 4 to smithi07929762-7 2024-04-24T01:56:20.534 INFO:tasks.rados.rados.0.smithi079.stdout:282: oid 7 updating version 0 to 53 2024-04-24T01:56:20.534 INFO:tasks.rados.rados.0.smithi079.stdout:282: oid 7 version 53 is already newer than 53 2024-04-24T01:56:20.535 INFO:tasks.rados.rados.0.smithi079.stdout:282: oid 7 version 53 is already newer than 53 2024-04-24T01:56:20.535 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 7 v 53 (ObjNum 164 snap 24 seq_num 164) dirty exists 2024-04-24T01:56:20.535 INFO:tasks.rados.rados.0.smithi079.stdout:282: left oid 7 (ObjNum 164 snap 24 seq_num 164) 2024-04-24T01:56:20.537 INFO:tasks.rados.rados.0.smithi079.stdout:281: finishing write tid 2 to smithi07929762-6 2024-04-24T01:56:20.541 INFO:tasks.rados.rados.0.smithi079.stdout:281: finishing write tid 3 to smithi07929762-6 2024-04-24T01:56:20.543 INFO:tasks.rados.rados.0.smithi079.stdout:281: finishing write tid 4 to smithi07929762-6 2024-04-24T01:56:20.543 INFO:tasks.rados.rados.0.smithi079.stdout:281: oid 6 updating version 0 to 62 2024-04-24T01:56:20.543 INFO:tasks.rados.rados.0.smithi079.stdout:281: oid 6 version 62 is already newer than 60 2024-04-24T01:56:20.544 INFO:tasks.rados.rados.0.smithi079.stdout:281: oid 6 version 62 is already newer than 62 2024-04-24T01:56:20.544 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 6 v 62 (ObjNum 163 snap 24 seq_num 163) dirty exists 2024-04-24T01:56:20.544 INFO:tasks.rados.rados.0.smithi079.stdout:281: left oid 6 (ObjNum 163 snap 24 seq_num 163) 2024-04-24T01:56:21.021 INFO:tasks.rados.rados.0.smithi079.stdout:278: done (4 left) 2024-04-24T01:56:21.021 INFO:tasks.rados.rados.0.smithi079.stdout:279: done (3 left) 2024-04-24T01:56:21.021 INFO:tasks.rados.rados.0.smithi079.stdout:281: done (2 left) 2024-04-24T01:56:21.021 INFO:tasks.rados.rados.0.smithi079.stdout:282: done (1 left) 2024-04-24T01:56:21.021 INFO:tasks.rados.rados.0.smithi079.stdout:283: done (0 left) 2024-04-24T01:56:21.021 INFO:tasks.rados.rados.0.smithi079.stdout:284: read oid 13 snap -1 2024-04-24T01:56:21.021 INFO:tasks.rados.rados.0.smithi079.stdout:284: expect deleted 2024-04-24T01:56:21.022 INFO:tasks.rados.rados.0.smithi079.stdout:285: snap_remove snap 24 2024-04-24T01:56:21.838 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T01:56:21.950 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:21.949+0000 7f5c28e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:21.954 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:21.954+0000 7f5c28e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:22.021 INFO:tasks.rados.rados.0.smithi079.stdout:285: done (1 left) 2024-04-24T01:56:22.022 INFO:tasks.rados.rados.0.smithi079.stdout:286: write oid 9 current snap is 25 2024-04-24T01:56:22.022 INFO:tasks.rados.rados.0.smithi079.stdout:286: seq_num 165 ranges {773155=459498,1722470=430045,2648676=124479} 2024-04-24T01:56:22.028 INFO:tasks.rados.rados.0.smithi079.stdout:286: writing smithi07929762-9 from 773155 to 1232653 tid 1 2024-04-24T01:56:22.033 INFO:tasks.rados.rados.0.smithi079.stdout:286: writing smithi07929762-9 from 1722470 to 2152515 tid 2 2024-04-24T01:56:22.037 INFO:tasks.rados.rados.0.smithi079.stdout:286: writing smithi07929762-9 from 2648676 to 2773155 tid 3 2024-04-24T01:56:22.037 INFO:tasks.rados.rados.0.smithi079.stdout:287: delete oid 26 current snap is 25 2024-04-24T01:56:22.103 INFO:tasks.rados.rados.0.smithi079.stdout:286: finishing write tid 1 to smithi07929762-9 2024-04-24T01:56:22.114 INFO:tasks.rados.rados.0.smithi079.stdout:284: done (2 left) 2024-04-24T01:56:22.115 INFO:tasks.rados.rados.0.smithi079.stdout:287: done (1 left) 2024-04-24T01:56:22.115 INFO:tasks.rados.rados.0.smithi079.stdout:288: read oid 6 snap 23 2024-04-24T01:56:22.115 INFO:tasks.rados.rados.0.smithi079.stdout:288: expect (ObjNum 66 snap 3 seq_num 66) 2024-04-24T01:56:22.115 INFO:tasks.rados.rados.0.smithi079.stdout:289: snap_create 2024-04-24T01:56:22.122 INFO:tasks.rados.rados.0.smithi079.stdout:286: finishing write tid 2 to smithi07929762-9 2024-04-24T01:56:22.129 INFO:tasks.rados.rados.0.smithi079.stdout:286: finishing write tid 3 to smithi07929762-9 2024-04-24T01:56:22.133 INFO:tasks.rados.rados.0.smithi079.stdout:286: finishing write tid 4 to smithi07929762-9 2024-04-24T01:56:22.133 INFO:tasks.rados.rados.0.smithi079.stdout:286: finishing write tid 5 to smithi07929762-9 2024-04-24T01:56:22.133 INFO:tasks.rados.rados.0.smithi079.stdout:286: oid 9 updating version 0 to 75 2024-04-24T01:56:22.134 INFO:tasks.rados.rados.0.smithi079.stdout:286: oid 9 version 75 is already newer than 75 2024-04-24T01:56:22.134 INFO:tasks.rados.rados.0.smithi079.stdout:286: oid 9 version 75 is already newer than 74 2024-04-24T01:56:22.134 INFO:tasks.rados.rados.0.smithi079.stdout:286: oid 9 version 75 is already newer than 75 2024-04-24T01:56:22.134 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 9 v 75 (ObjNum 165 snap 25 seq_num 165) dirty exists 2024-04-24T01:56:22.134 INFO:tasks.rados.rados.0.smithi079.stdout:286: left oid 9 (ObjNum 165 snap 25 seq_num 165) 2024-04-24T01:56:22.187 INFO:tasks.rados.rados.0.smithi079.stdout:288: expect (ObjNum 66 snap 3 seq_num 66) 2024-04-24T01:56:22.516 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-04-24T01:56:22.516 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-04-24T01:56:22.626 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:22.619+0000 7f99d046d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:22.629 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:22.628+0000 7f99d046d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:23.018 INFO:teuthology.orchestra.run.smithi079.stderr:reweighted osd.0 to 1 (10000) 2024-04-24T01:56:23.022 INFO:tasks.rados.rados.0.smithi079.stdout:286: done (2 left) 2024-04-24T01:56:23.023 INFO:tasks.rados.rados.0.smithi079.stdout:288: done (1 left) 2024-04-24T01:56:23.023 INFO:tasks.rados.rados.0.smithi079.stdout:289: done (0 left) 2024-04-24T01:56:23.023 INFO:tasks.rados.rados.0.smithi079.stdout:290: write (excl) oid 7 current snap is 26 2024-04-24T01:56:23.023 INFO:tasks.rados.rados.0.smithi079.stdout:290: seq_num 167 ranges {406769=609996,1439434=433714,2303522=503247} 2024-04-24T01:56:23.028 INFO:tasks.rados.rados.0.smithi079.stdout:290: writing smithi07929762-7 from 406769 to 1016765 tid 1 2024-04-24T01:56:23.033 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-04-24T01:56:23.034 INFO:tasks.rados.rados.0.smithi079.stdout:290: writing smithi07929762-7 from 1439434 to 1873148 tid 2 2024-04-24T01:56:23.039 INFO:tasks.rados.rados.0.smithi079.stdout:290: writing smithi07929762-7 from 2303522 to 2806769 tid 3 2024-04-24T01:56:23.039 INFO:tasks.rados.rados.0.smithi079.stdout:291: read oid 50 snap -1 2024-04-24T01:56:23.039 INFO:tasks.rados.rados.0.smithi079.stdout:291: expect (ObjNum 162 snap 24 seq_num 162) 2024-04-24T01:56:23.039 INFO:tasks.rados.rados.0.smithi079.stdout:292: read oid 36 snap -1 2024-04-24T01:56:23.039 INFO:tasks.rados.rados.0.smithi079.stdout:292: expect deleted 2024-04-24T01:56:23.040 INFO:tasks.rados.rados.0.smithi079.stdout:293: read oid 20 snap 23 2024-04-24T01:56:23.040 INFO:tasks.rados.rados.0.smithi079.stdout:293: expect (ObjNum 154 snap 21 seq_num 154) 2024-04-24T01:56:23.040 INFO:tasks.rados.rados.0.smithi079.stdout:294: delete oid 23 current snap is 26 2024-04-24T01:56:23.082 INFO:tasks.rados.rados.0.smithi079.stdout:293: expect (ObjNum 154 snap 21 seq_num 154) 2024-04-24T01:56:23.179 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:23.172+0000 7f7cbaf1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:23.183 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:23.182+0000 7f7cbaf1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:23.249 INFO:tasks.rados.rados.0.smithi079.stdout:290: finishing write tid 1 to smithi07929762-7 2024-04-24T01:56:23.249 INFO:tasks.rados.rados.0.smithi079.stdout:291: expect (ObjNum 162 snap 24 seq_num 162) 2024-04-24T01:56:23.414 INFO:tasks.rados.rados.0.smithi079.stdout:290: finishing write tid 2 to smithi07929762-7 2024-04-24T01:56:23.414 INFO:tasks.rados.rados.0.smithi079.stdout:290: finishing write tid 3 to smithi07929762-7 2024-04-24T01:56:23.414 INFO:tasks.rados.rados.0.smithi079.stdout:290: finishing write tid 4 to smithi07929762-7 2024-04-24T01:56:23.414 INFO:tasks.rados.rados.0.smithi079.stdout:290: finishing write tid 5 to smithi07929762-7 2024-04-24T01:56:23.415 INFO:tasks.rados.rados.0.smithi079.stdout:290: oid 7 updating version 0 to 58 2024-04-24T01:56:23.415 INFO:tasks.rados.rados.0.smithi079.stdout:290: oid 7 version 58 is already newer than 55 2024-04-24T01:56:23.415 INFO:tasks.rados.rados.0.smithi079.stdout:290: oid 7 version 58 is already newer than 58 2024-04-24T01:56:23.415 INFO:tasks.rados.rados.0.smithi079.stdout:290: oid 7 version 58 is already newer than 58 2024-04-24T01:56:23.415 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 7 v 58 (ObjNum 167 snap 26 seq_num 167) dirty exists 2024-04-24T01:56:23.415 INFO:tasks.rados.rados.0.smithi079.stdout:290: left oid 7 (ObjNum 167 snap 26 seq_num 167) 2024-04-24T01:56:23.415 INFO:tasks.rados.rados.0.smithi079.stdout:290: done (4 left) 2024-04-24T01:56:23.415 INFO:tasks.rados.rados.0.smithi079.stdout:291: done (3 left) 2024-04-24T01:56:23.415 INFO:tasks.rados.rados.0.smithi079.stdout:292: done (2 left) 2024-04-24T01:56:23.415 INFO:tasks.rados.rados.0.smithi079.stdout:293: done (1 left) 2024-04-24T01:56:23.415 INFO:tasks.rados.rados.0.smithi079.stdout:294: done (0 left) 2024-04-24T01:56:23.415 INFO:tasks.rados.rados.0.smithi079.stdout:295: write oid 29 current snap is 26 2024-04-24T01:56:23.415 INFO:tasks.rados.rados.0.smithi079.stdout:295: seq_num 169 ranges {440383=476847,1556398=437383,2474721=365662} 2024-04-24T01:56:23.419 INFO:tasks.rados.rados.0.smithi079.stdout:295: writing smithi07929762-29 from 440383 to 917230 tid 1 2024-04-24T01:56:23.426 INFO:tasks.rados.rados.0.smithi079.stdout:295: writing smithi07929762-29 from 1556398 to 1993781 tid 2 2024-04-24T01:56:23.434 INFO:tasks.rados.rados.0.smithi079.stdout:295: writing smithi07929762-29 from 2474721 to 2840383 tid 3 2024-04-24T01:56:23.435 INFO:tasks.rados.rados.0.smithi079.stdout:296: write oid 28 current snap is 26 2024-04-24T01:56:23.435 INFO:tasks.rados.rados.0.smithi079.stdout:296: seq_num 170 ranges {457190=552096,1614880=581041,2785791=71399} 2024-04-24T01:56:23.440 INFO:tasks.rados.rados.0.smithi079.stdout:296: writing smithi07929762-28 from 457190 to 1009286 tid 1 2024-04-24T01:56:23.447 INFO:tasks.rados.rados.0.smithi079.stdout:296: writing smithi07929762-28 from 1614880 to 2195921 tid 2 2024-04-24T01:56:23.451 INFO:tasks.rados.rados.0.smithi079.stdout:296: writing smithi07929762-28 from 2785791 to 2857190 tid 3 2024-04-24T01:56:23.451 INFO:tasks.rados.rados.0.smithi079.stdout:297: snap_create 2024-04-24T01:56:23.486 INFO:tasks.rados.rados.0.smithi079.stdout:295: finishing write tid 1 to smithi07929762-29 2024-04-24T01:56:23.510 INFO:tasks.rados.rados.0.smithi079.stdout:295: finishing write tid 2 to smithi07929762-29 2024-04-24T01:56:23.513 INFO:tasks.rados.rados.0.smithi079.stdout:295: finishing write tid 3 to smithi07929762-29 2024-04-24T01:56:23.517 INFO:tasks.rados.rados.0.smithi079.stdout:296: finishing write tid 1 to smithi07929762-28 2024-04-24T01:56:23.518 INFO:tasks.rados.rados.0.smithi079.stdout:295: finishing write tid 4 to smithi07929762-29 2024-04-24T01:56:23.520 INFO:tasks.rados.rados.0.smithi079.stdout:295: finishing write tid 5 to smithi07929762-29 2024-04-24T01:56:23.521 INFO:tasks.rados.rados.0.smithi079.stdout:295: oid 29 updating version 0 to 59 2024-04-24T01:56:23.521 INFO:tasks.rados.rados.0.smithi079.stdout:295: oid 29 version 59 is already newer than 59 2024-04-24T01:56:23.521 INFO:tasks.rados.rados.0.smithi079.stdout:295: oid 29 version 59 is already newer than 59 2024-04-24T01:56:23.521 INFO:tasks.rados.rados.0.smithi079.stdout:295: oid 29 version 59 is already newer than 59 2024-04-24T01:56:23.521 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 29 v 59 (ObjNum 169 snap 26 seq_num 169) dirty exists 2024-04-24T01:56:23.521 INFO:tasks.rados.rados.0.smithi079.stdout:295: left oid 29 (ObjNum 169 snap 26 seq_num 169) 2024-04-24T01:56:23.529 INFO:tasks.rados.rados.0.smithi079.stdout:296: finishing write tid 2 to smithi07929762-28 2024-04-24T01:56:23.534 INFO:tasks.rados.rados.0.smithi079.stdout:296: finishing write tid 3 to smithi07929762-28 2024-04-24T01:56:23.538 INFO:tasks.rados.rados.0.smithi079.stdout:296: finishing write tid 4 to smithi07929762-28 2024-04-24T01:56:23.539 INFO:tasks.rados.rados.0.smithi079.stdout:296: finishing write tid 5 to smithi07929762-28 2024-04-24T01:56:23.539 INFO:tasks.rados.rados.0.smithi079.stdout:296: oid 28 updating version 0 to 80 2024-04-24T01:56:23.540 INFO:tasks.rados.rados.0.smithi079.stdout:296: oid 28 version 80 is already newer than 80 2024-04-24T01:56:23.540 INFO:tasks.rados.rados.0.smithi079.stdout:296: oid 28 version 80 is already newer than 80 2024-04-24T01:56:23.540 INFO:tasks.rados.rados.0.smithi079.stdout:296: oid 28 version 80 is already newer than 80 2024-04-24T01:56:23.540 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 28 v 80 (ObjNum 170 snap 26 seq_num 170) dirty exists 2024-04-24T01:56:23.540 INFO:tasks.rados.rados.0.smithi079.stdout:296: left oid 28 (ObjNum 170 snap 26 seq_num 170) 2024-04-24T01:56:24.018 INFO:teuthology.orchestra.run.smithi079.stderr:reweighted osd.1 to 1 (10000) 2024-04-24T01:56:24.022 INFO:tasks.rados.rados.0.smithi079.stdout:295: done (2 left) 2024-04-24T01:56:24.022 INFO:tasks.rados.rados.0.smithi079.stdout:296: done (1 left) 2024-04-24T01:56:24.022 INFO:tasks.rados.rados.0.smithi079.stdout:297: done (0 left) 2024-04-24T01:56:24.022 INFO:tasks.rados.rados.0.smithi079.stdout:298: read oid 24 snap 25 2024-04-24T01:56:24.022 INFO:tasks.rados.rados.0.smithi079.stdout:298: expect (ObjNum 56 snap 0 seq_num 56) 2024-04-24T01:56:24.022 INFO:tasks.rados.rados.0.smithi079.stdout:299: snap_remove snap 23 2024-04-24T01:56:24.036 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-04-24T01:56:24.051 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T01:56:24.189 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:24.187+0000 7fb81026d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:24.193 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:24.193+0000 7fb81026d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:24.205 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:24.198+0000 7f6fd52ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:24.209 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:24.209+0000 7f6fd52ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:25.022 INFO:teuthology.orchestra.run.smithi079.stderr:reweighted osd.2 to 1 (10000) 2024-04-24T01:56:25.025 INFO:tasks.rados.rados.0.smithi079.stdout:299: done (1 left) 2024-04-24T01:56:25.025 INFO:tasks.rados.rados.0.smithi079.stdout:300: read oid 6 snap -1 2024-04-24T01:56:25.025 INFO:tasks.rados.rados.0.smithi079.stdout:300: expect (ObjNum 163 snap 24 seq_num 163) 2024-04-24T01:56:25.026 INFO:tasks.rados.rados.0.smithi079.stdout:301: snap_remove snap 26 2024-04-24T01:56:25.038 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-04-24T01:56:25.165 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:25.154+0000 7f4e95b5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:25.171 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:25.170+0000 7f4e95b5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:26.017 INFO:teuthology.orchestra.run.smithi079.stderr:reweighted osd.3 to 1 (10000) 2024-04-24T01:56:26.019 INFO:tasks.rados.rados.0.smithi079.stdout:301: done (2 left) 2024-04-24T01:56:26.019 INFO:tasks.rados.rados.0.smithi079.stdout:302: write oid 31 current snap is 27 2024-04-24T01:56:26.019 INFO:tasks.rados.rados.0.smithi079.stdout:302: seq_num 171 ranges {473997=627345,1673362=441052,2529567=344430} 2024-04-24T01:56:26.027 INFO:tasks.rados.rados.0.smithi079.stdout:302: writing smithi07929762-31 from 473997 to 1101342 tid 1 2024-04-24T01:56:26.031 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-04-24T01:56:26.032 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T01:56:26.034 INFO:tasks.rados.rados.0.smithi079.stdout:302: writing smithi07929762-31 from 1673362 to 2114414 tid 2 2024-04-24T01:56:26.038 INFO:tasks.rados.rados.0.smithi079.stdout:302: writing smithi07929762-31 from 2529567 to 2873997 tid 3 2024-04-24T01:56:26.038 INFO:tasks.rados.rados.0.smithi079.stdout:303: read oid 39 snap -1 2024-04-24T01:56:26.038 INFO:tasks.rados.rados.0.smithi079.stdout:303: expect deleted 2024-04-24T01:56:26.038 INFO:tasks.rados.rados.0.smithi079.stdout:304: write (excl) oid 17 current snap is 27 2024-04-24T01:56:26.038 INFO:tasks.rados.rados.0.smithi079.stdout:304: seq_num 172 ranges {490804=702594,1615491=584710,2724284=166520} 2024-04-24T01:56:26.047 INFO:tasks.rados.rados.0.smithi079.stdout:304: writing smithi07929762-17 from 490804 to 1193398 tid 1 2024-04-24T01:56:26.055 INFO:tasks.rados.rados.0.smithi079.stdout:304: writing smithi07929762-17 from 1615491 to 2200201 tid 2 2024-04-24T01:56:26.060 INFO:tasks.rados.rados.0.smithi079.stdout:304: writing smithi07929762-17 from 2724284 to 2890804 tid 3 2024-04-24T01:56:26.061 INFO:tasks.rados.rados.0.smithi079.stdout:305: snap_create 2024-04-24T01:56:26.061 INFO:tasks.rados.rados.0.smithi079.stdout:298: expect (ObjNum 56 snap 0 seq_num 56) 2024-04-24T01:56:26.166 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:26.160+0000 7f903915d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:26.173 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:26.172+0000 7f903915d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:26.229 INFO:tasks.rados.rados.0.smithi079.stdout:300: expect (ObjNum 163 snap 24 seq_num 163) 2024-04-24T01:56:26.295 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T01:56:26.410 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:56:26.410 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-24T01:56:26.422 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:26.420+0000 7feceeb9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:26.423 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":99,"stamp":"2024-04-24T01:56:26.017889+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":91196542,"num_objects":52,"num_object_clones":12,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":746,"num_read_kb":387764,"num_write":745,"num_write_kb":166700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":839,"ondisk_log_size":839,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":308160,"kb_used_data":200864,"kb_used_omap":0,"kb_used_meta":107264,"kb_avail":377179200,"statfs":{"total":386547056640,"available":386231500800,"internally_reserved":0,"allocated":205684736,"data_stored":219237753,"data_compressed":89889,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109838336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-8134456,"num_objects":1,"num_object_clones":2,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":36,"num_read_kb":25605,"num_write":34,"num_write_kb":8450,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.997735"},"pg_stats":[{"pgid":"3.f","version":"67'26","reported_seq":211,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-04-24T01:56:24.149820+0000","last_change":"2024-04-24T01:56:22.247006+0000","last_active":"2024-04-24T01:56:24.149820+0000","last_peered":"2024-04-24T01:56:24.149820+0000","last_clean":"2024-04-24T01:56:24.149820+0000","last_became_active":"2024-04-24T01:55:30.386726+0000","last_became_peered":"2024-04-24T01:55:30.386726+0000","last_unstale":"2024-04-24T01:56:24.149820+0000","last_undegraded":"2024-04-24T01:56:24.149820+0000","last_fullsized":"2024-04-24T01:56:24.149820+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0,"stat_sum":{"num_bytes":2437015,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":38,"num_read_kb":28311,"num_write":17,"num_write_kb":5216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"69'80","reported_seq":259,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-04-24T01:56:24.136941+0000","last_change":"2024-04-24T01:56:22.150144+0000","last_active":"2024-04-24T01:56:24.136941+0000","last_peered":"2024-04-24T01:56:24.136941+0000","last_clean":"2024-04-24T01:56:24.136941+0000","last_became_active":"2024-04-24T01:55:30.321866+0000","last_became_peered":"2024-04-24T01:55:30.321866+0000","last_unstale":"2024-04-24T01:56:24.136941+0000","last_undegraded":"2024-04-24T01:56:24.136941+0000","last_fullsized":"2024-04-24T01:56:24.136941+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0,"stat_sum":{"num_bytes":10218623,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":51,"num_read_kb":25527,"num_write":61,"num_write_kb":16505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"67'54","reported_seq":226,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-04-24T01:56:23.040045+0000","last_change":"2024-04-24T01:56:22.052276+0000","last_active":"2024-04-24T01:56:23.040045+0000","last_peered":"2024-04-24T01:56:23.040045+0000","last_clean":"2024-04-24T01:56:23.040045+0000","last_became_active":"2024-04-24T01:55:30.320186+0000","last_became_peered":"2024-04-24T01:55:30.320186+0000","last_unstale":"2024-04-24T01:56:23.040045+0000","last_undegraded":"2024-04-24T01:56:23.040045+0000","last_fullsized":"2024-04-24T01:56:23.040045+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":5,"snaptrim_duration":0,"stat_sum":{"num_bytes":5882417,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":26,"num_read_kb":8133,"num_write":48,"num_write_kb":12133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"67'65","reported_seq":249,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-04-24T01:56:24.145324+0000","last_change":"2024-04-24T01:56:22.246581+0000","last_active":"2024-04-24T01:56:24.145324+0000","last_peered":"2024-04-24T01:56:24.145324+0000","last_clean":"2024-04-24T01:56:24.145324+0000","last_became_active":"2024-04-24T01:55:30.288081+0000","last_became_peered":"2024-04-24T01:55:30.288081+0000","last_unstale":"2024-04-24T01:56:24.145324+0000","last_undegraded":"2024-04-24T01:56:24.145324+0000","last_fullsized":"2024-04-24T01:56:24.145324+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":8,"snaptrim_duration":0,"stat_sum":{"num_bytes":7395047,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":60,"num_read_kb":44504,"num_write":41,"num_write_kb":11889,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"39'23","reported_seq":207,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-04-24T01:56:24.047501+0000","last_change":"2024-04-24T01:56:22.258981+0000","last_active":"2024-04-24T01:56:24.047501+0000","last_peered":"2024-04-24T01:56:24.047501+0000","last_clean":"2024-04-24T01:56:24.047501+0000","last_became_active":"2024-04-24T01:55:30.333815+0000","last_became_peered":"2024-04-24T01:55:30.333815+0000","last_unstale":"2024-04-24T01:56:24.047501+0000","last_undegraded":"2024-04-24T01:56:24.047501+0000","last_fullsized":"2024-04-24T01:56:24.047501+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0,"stat_sum":{"num_bytes":3243718,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":28,"num_read_kb":17554,"num_write":25,"num_write_kb":6861,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"67'93","reported_seq":294,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-04-24T01:56:24.148538+0000","last_change":"2024-04-24T01:56:22.247349+0000","last_active":"2024-04-24T01:56:24.148538+0000","last_peered":"2024-04-24T01:56:24.148538+0000","last_clean":"2024-04-24T01:56:24.148538+0000","last_became_active":"2024-04-24T01:55:30.315075+0000","last_became_peered":"2024-04-24T01:55:30.315075+0000","last_unstale":"2024-04-24T01:56:24.148538+0000","last_undegraded":"2024-04-24T01:56:24.148538+0000","last_fullsized":"2024-04-24T01:56:24.148538+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":11,"snaptrim_duration":0,"stat_sum":{"num_bytes":14672445,"num_objects":7,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":90,"num_read_kb":52955,"num_write":83,"num_write_kb":22850,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"69'19","reported_seq":194,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-04-24T01:56:24.154081+0000","last_change":"2024-04-24T01:56:22.253377+0000","last_active":"2024-04-24T01:56:24.154081+0000","last_peered":"2024-04-24T01:56:24.154081+0000","last_clean":"2024-04-24T01:56:24.154081+0000","last_became_active":"2024-04-24T01:55:30.350300+0000","last_became_peered":"2024-04-24T01:55:30.350300+0000","last_unstale":"2024-04-24T01:56:24.154081+0000","last_undegraded":"2024-04-24T01:56:24.154081+0000","last_fullsized":"2024-04-24T01:56:24.154081+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":5003,"num_write":15,"num_write_kb":3526,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"64'6","reported_seq":174,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-04-24T01:56:23.048519+0000","last_change":"2024-04-24T01:56:22.055418+0000","last_active":"2024-04-24T01:56:23.048519+0000","last_peered":"2024-04-24T01:56:23.048519+0000","last_clean":"2024-04-24T01:56:23.048519+0000","last_became_active":"2024-04-24T01:55:30.376813+0000","last_became_peered":"2024-04-24T01:55:30.376813+0000","last_unstale":"2024-04-24T01:56:23.048519+0000","last_undegraded":"2024-04-24T01:56:23.048519+0000","last_fullsized":"2024-04-24T01:56:23.048519+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'97","reported_seq":290,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-04-24T01:56:24.039491+0000","last_change":"2024-04-24T01:55:16.196470+0000","last_active":"2024-04-24T01:56:24.039491+0000","last_peered":"2024-04-24T01:56:24.039491+0000","last_clean":"2024-04-24T01:56:24.039491+0000","last_became_active":"2024-04-24T01:55:16.190464+0000","last_became_peered":"2024-04-24T01:55:16.190464+0000","last_unstale":"2024-04-24T01:56:24.039491+0000","last_undegraded":"2024-04-24T01:56:24.039491+0000","last_fullsized":"2024-04-24T01:56:24.039491+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:15.127115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:15.127115+0000","last_clean_scrub_stamp":"2024-04-24T01:55:15.127115+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":154,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-04-24T01:56:23.048975+0000","last_change":"2024-04-24T01:55:18.236148+0000","last_active":"2024-04-24T01:56:23.048975+0000","last_peered":"2024-04-24T01:56:23.048975+0000","last_clean":"2024-04-24T01:56:23.048975+0000","last_became_active":"2024-04-24T01:55:18.230154+0000","last_became_peered":"2024-04-24T01:55:18.230154+0000","last_unstale":"2024-04-24T01:56:23.048975+0000","last_undegraded":"2024-04-24T01:56:23.048975+0000","last_fullsized":"2024-04-24T01:56:23.048975+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"68'66","reported_seq":260,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-04-24T01:56:23.049408+0000","last_change":"2024-04-24T01:56:22.054610+0000","last_active":"2024-04-24T01:56:23.049408+0000","last_peered":"2024-04-24T01:56:23.049408+0000","last_clean":"2024-04-24T01:56:23.049408+0000","last_became_active":"2024-04-24T01:55:30.316222+0000","last_became_peered":"2024-04-24T01:55:30.316222+0000","last_unstale":"2024-04-24T01:56:23.049408+0000","last_undegraded":"2024-04-24T01:56:23.049408+0000","last_fullsized":"2024-04-24T01:56:23.049408+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0,"stat_sum":{"num_bytes":4487469,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":74,"num_read_kb":46242,"num_write":56,"num_write_kb":15303,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":159,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-04-24T01:56:24.156239+0000","last_change":"2024-04-24T01:55:18.265905+0000","last_active":"2024-04-24T01:56:24.156239+0000","last_peered":"2024-04-24T01:56:24.156239+0000","last_clean":"2024-04-24T01:56:24.156239+0000","last_became_active":"2024-04-24T01:55:18.257020+0000","last_became_peered":"2024-04-24T01:55:18.257020+0000","last_unstale":"2024-04-24T01:56:24.156239+0000","last_undegraded":"2024-04-24T01:56:24.156239+0000","last_fullsized":"2024-04-24T01:56:24.156239+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"52'34","reported_seq":223,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-04-24T01:56:24.073359+0000","last_change":"2024-04-24T01:56:22.275985+0000","last_active":"2024-04-24T01:56:24.073359+0000","last_peered":"2024-04-24T01:56:24.073359+0000","last_clean":"2024-04-24T01:56:24.073359+0000","last_became_active":"2024-04-24T01:55:30.344129+0000","last_became_peered":"2024-04-24T01:55:30.344129+0000","last_unstale":"2024-04-24T01:56:24.073359+0000","last_undegraded":"2024-04-24T01:56:24.073359+0000","last_fullsized":"2024-04-24T01:56:24.073359+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0,"stat_sum":{"num_bytes":9109328,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":44,"num_read_kb":25529,"num_write":32,"num_write_kb":8371,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":154,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-04-24T01:56:23.032842+0000","last_change":"2024-04-24T01:55:18.215251+0000","last_active":"2024-04-24T01:56:23.032842+0000","last_peered":"2024-04-24T01:56:23.032842+0000","last_clean":"2024-04-24T01:56:23.032842+0000","last_became_active":"2024-04-24T01:55:18.206605+0000","last_became_peered":"2024-04-24T01:55:18.206605+0000","last_unstale":"2024-04-24T01:56:23.032842+0000","last_undegraded":"2024-04-24T01:56:23.032842+0000","last_fullsized":"2024-04-24T01:56:23.032842+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"39'22","reported_seq":199,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-04-24T01:56:24.045955+0000","last_change":"2024-04-24T01:56:22.260834+0000","last_active":"2024-04-24T01:56:24.045955+0000","last_peered":"2024-04-24T01:56:24.045955+0000","last_clean":"2024-04-24T01:56:24.045955+0000","last_became_active":"2024-04-24T01:55:30.401366+0000","last_became_peered":"2024-04-24T01:55:30.401366+0000","last_unstale":"2024-04-24T01:56:24.045955+0000","last_undegraded":"2024-04-24T01:56:24.045955+0000","last_fullsized":"2024-04-24T01:56:24.045955+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0,"stat_sum":{"num_bytes":7042067,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":9364,"num_write":18,"num_write_kb":5026,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":159,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-04-24T01:56:24.150672+0000","last_change":"2024-04-24T01:55:18.230893+0000","last_active":"2024-04-24T01:56:24.150672+0000","last_peered":"2024-04-24T01:56:24.150672+0000","last_clean":"2024-04-24T01:56:24.150672+0000","last_became_active":"2024-04-24T01:55:18.225188+0000","last_became_peered":"2024-04-24T01:55:18.225188+0000","last_unstale":"2024-04-24T01:56:24.150672+0000","last_undegraded":"2024-04-24T01:56:24.150672+0000","last_fullsized":"2024-04-24T01:56:24.150672+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"69'58","reported_seq":236,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-04-24T01:56:24.139134+0000","last_change":"2024-04-24T01:56:22.150445+0000","last_active":"2024-04-24T01:56:24.139134+0000","last_peered":"2024-04-24T01:56:24.139134+0000","last_clean":"2024-04-24T01:56:24.139134+0000","last_became_active":"2024-04-24T01:55:30.343318+0000","last_became_peered":"2024-04-24T01:55:30.343318+0000","last_unstale":"2024-04-24T01:56:24.139134+0000","last_undegraded":"2024-04-24T01:56:24.139134+0000","last_fullsized":"2024-04-24T01:56:24.139134+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":3,"snaptrim_duration":0,"stat_sum":{"num_bytes":6571504,"num_objects":6,"num_object_clones":3,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":41,"num_read_kb":28772,"num_write":46,"num_write_kb":13987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"68'75","reported_seq":262,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-04-24T01:56:23.080803+0000","last_change":"2024-04-24T01:56:22.097786+0000","last_active":"2024-04-24T01:56:23.080803+0000","last_peered":"2024-04-24T01:56:23.080803+0000","last_clean":"2024-04-24T01:56:23.080803+0000","last_became_active":"2024-04-24T01:55:30.345308+0000","last_became_peered":"2024-04-24T01:55:30.345308+0000","last_unstale":"2024-04-24T01:56:23.080803+0000","last_undegraded":"2024-04-24T01:56:23.080803+0000","last_fullsized":"2024-04-24T01:56:23.080803+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":7,"snaptrim_duration":0,"stat_sum":{"num_bytes":9495955,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":71,"num_read_kb":40506,"num_write":58,"num_write_kb":15036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":155,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-04-24T01:56:24.143486+0000","last_change":"2024-04-24T01:55:18.224238+0000","last_active":"2024-04-24T01:56:24.143486+0000","last_peered":"2024-04-24T01:56:24.143486+0000","last_clean":"2024-04-24T01:56:24.143486+0000","last_became_active":"2024-04-24T01:55:18.219131+0000","last_became_peered":"2024-04-24T01:55:18.219131+0000","last_unstale":"2024-04-24T01:56:24.143486+0000","last_undegraded":"2024-04-24T01:56:24.143486+0000","last_fullsized":"2024-04-24T01:56:24.143486+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"66'31","reported_seq":207,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-04-24T01:56:24.143805+0000","last_change":"2024-04-24T01:56:22.162185+0000","last_active":"2024-04-24T01:56:24.143805+0000","last_peered":"2024-04-24T01:56:24.143805+0000","last_clean":"2024-04-24T01:56:24.143805+0000","last_became_active":"2024-04-24T01:55:30.319958+0000","last_became_peered":"2024-04-24T01:55:30.319958+0000","last_unstale":"2024-04-24T01:56:24.143805+0000","last_undegraded":"2024-04-24T01:56:24.143805+0000","last_fullsized":"2024-04-24T01:56:24.143805+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0,"stat_sum":{"num_bytes":2638699,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":21,"num_read_kb":19606,"num_write":27,"num_write_kb":7720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":157,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-04-24T01:56:24.045824+0000","last_change":"2024-04-24T01:55:18.235763+0000","last_active":"2024-04-24T01:56:24.045824+0000","last_peered":"2024-04-24T01:56:24.045824+0000","last_clean":"2024-04-24T01:56:24.045824+0000","last_became_active":"2024-04-24T01:55:18.229175+0000","last_became_peered":"2024-04-24T01:55:18.229175+0000","last_unstale":"2024-04-24T01:56:24.045824+0000","last_undegraded":"2024-04-24T01:56:24.045824+0000","last_fullsized":"2024-04-24T01:56:24.045824+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"68'28","reported_seq":207,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-04-24T01:56:24.045669+0000","last_change":"2024-04-24T01:56:23.063637+0000","last_active":"2024-04-24T01:56:24.045669+0000","last_peered":"2024-04-24T01:56:24.045669+0000","last_clean":"2024-04-24T01:56:24.045669+0000","last_became_active":"2024-04-24T01:55:30.372690+0000","last_became_peered":"2024-04-24T01:55:30.372690+0000","last_unstale":"2024-04-24T01:56:24.045669+0000","last_undegraded":"2024-04-24T01:56:24.045669+0000","last_fullsized":"2024-04-24T01:56:24.045669+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":3,"snaptrim_duration":0,"stat_sum":{"num_bytes":2521050,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":28,"num_read_kb":16701,"num_write":24,"num_write_kb":6609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":155,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-04-24T01:56:24.146300+0000","last_change":"2024-04-24T01:55:18.278351+0000","last_active":"2024-04-24T01:56:24.146300+0000","last_peered":"2024-04-24T01:56:24.146300+0000","last_clean":"2024-04-24T01:56:24.146300+0000","last_became_active":"2024-04-24T01:55:18.273668+0000","last_became_peered":"2024-04-24T01:55:18.273668+0000","last_unstale":"2024-04-24T01:56:24.146300+0000","last_undegraded":"2024-04-24T01:56:24.146300+0000","last_fullsized":"2024-04-24T01:56:24.146300+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"66'62","reported_seq":239,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-04-24T01:56:24.144972+0000","last_change":"2024-04-24T01:56:22.162495+0000","last_active":"2024-04-24T01:56:24.144972+0000","last_peered":"2024-04-24T01:56:24.144972+0000","last_clean":"2024-04-24T01:56:24.144972+0000","last_became_active":"2024-04-24T01:55:30.363123+0000","last_became_peered":"2024-04-24T01:55:30.363123+0000","last_unstale":"2024-04-24T01:56:24.144972+0000","last_undegraded":"2024-04-24T01:56:24.144972+0000","last_fullsized":"2024-04-24T01:56:24.144972+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0,"stat_sum":{"num_bytes":4890837,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":39,"num_read_kb":18965,"num_write":52,"num_write_kb":12926,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":154,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-04-24T01:56:23.046914+0000","last_change":"2024-04-24T01:55:18.241117+0000","last_active":"2024-04-24T01:56:23.046914+0000","last_peered":"2024-04-24T01:56:23.046914+0000","last_clean":"2024-04-24T01:56:23.046914+0000","last_became_active":"2024-04-24T01:55:18.234313+0000","last_became_peered":"2024-04-24T01:55:18.234313+0000","last_unstale":"2024-04-24T01:56:23.046914+0000","last_undegraded":"2024-04-24T01:56:23.046914+0000","last_fullsized":"2024-04-24T01:56:23.046914+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":90606174,"num_objects":50,"num_object_clones":12,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":643,"num_read_kb":387673,"num_write":608,"num_write_kb":164806,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":205959168,"data_stored":222057788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":742,"ondisk_log_size":742,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":1}],"osd_stats":[{"osd":3,"up_from":10,"seq":42949673029,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":83820,"kb_used_data":56996,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94288020,"statfs":{"total":96636764160,"available":96550932480,"internally_reserved":0,"allocated":58363904,"data_stored":62196776,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673029,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":74720,"kb_used_data":47896,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94297120,"statfs":{"total":96636764160,"available":96560250880,"internally_reserved":0,"allocated":49045504,"data_stored":53847384,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705734,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":73836,"kb_used_data":47012,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94298004,"statfs":{"total":96636764160,"available":96561156096,"internally_reserved":0,"allocated":48140288,"data_stored":51675912,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705733,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":75784,"kb_used_data":48960,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94296056,"statfs":{"total":96636764160,"available":96559161344,"internally_reserved":0,"allocated":50135040,"data_stored":51517681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":48799744,"data_stored":52957136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":47288320,"data_stored":49019425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49631232,"data_stored":54815627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":60239872,"data_stored":65265600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T01:56:26.424 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T01:56:26.426 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:26.425+0000 7feceeb9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:26.432 INFO:tasks.rados.rados.0.smithi079.stdout:304: finishing write tid 1 to smithi07929762-17 2024-04-24T01:56:26.432 INFO:tasks.rados.rados.0.smithi079.stdout:304: finishing write tid 2 to smithi07929762-17 2024-04-24T01:56:26.432 INFO:tasks.rados.rados.0.smithi079.stdout:304: finishing write tid 3 to smithi07929762-17 2024-04-24T01:56:26.432 INFO:tasks.rados.rados.0.smithi079.stdout:304: finishing write tid 4 to smithi07929762-17 2024-04-24T01:56:26.432 INFO:tasks.rados.rados.0.smithi079.stdout:304: finishing write tid 5 to smithi07929762-17 2024-04-24T01:56:26.432 INFO:tasks.rados.rados.0.smithi079.stdout:304: oid 17 updating version 0 to 68 2024-04-24T01:56:26.432 INFO:tasks.rados.rados.0.smithi079.stdout:304: oid 17 version 68 is already newer than 65 2024-04-24T01:56:26.432 INFO:tasks.rados.rados.0.smithi079.stdout:304: oid 17 version 68 is already newer than 68 2024-04-24T01:56:26.433 INFO:tasks.rados.rados.0.smithi079.stdout:304: oid 17 version 68 is already newer than 68 2024-04-24T01:56:26.433 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 17 v 68 (ObjNum 172 snap 27 seq_num 172) dirty exists 2024-04-24T01:56:26.433 INFO:tasks.rados.rados.0.smithi079.stdout:304: left oid 17 (ObjNum 172 snap 27 seq_num 172) 2024-04-24T01:56:26.433 INFO:tasks.rados.rados.0.smithi079.stdout:302: finishing write tid 1 to smithi07929762-31 2024-04-24T01:56:26.433 INFO:tasks.rados.rados.0.smithi079.stdout:302: finishing write tid 2 to smithi07929762-31 2024-04-24T01:56:26.433 INFO:tasks.rados.rados.0.smithi079.stdout:302: finishing write tid 3 to smithi07929762-31 2024-04-24T01:56:26.433 INFO:tasks.rados.rados.0.smithi079.stdout:302: finishing write tid 4 to smithi07929762-31 2024-04-24T01:56:26.433 INFO:tasks.rados.rados.0.smithi079.stdout:302: finishing write tid 5 to smithi07929762-31 2024-04-24T01:56:26.433 INFO:tasks.rados.rados.0.smithi079.stdout:302: oid 31 updating version 0 to 100 2024-04-24T01:56:26.433 INFO:tasks.rados.rados.0.smithi079.stdout:302: oid 31 version 100 is already newer than 97 2024-04-24T01:56:26.433 INFO:tasks.rados.rados.0.smithi079.stdout:302: oid 31 version 100 is already newer than 100 2024-04-24T01:56:26.433 INFO:tasks.rados.rados.0.smithi079.stdout:302: oid 31 version 100 is already newer than 100 2024-04-24T01:56:26.433 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 31 v 100 (ObjNum 171 snap 27 seq_num 171) dirty exists 2024-04-24T01:56:26.433 INFO:tasks.rados.rados.0.smithi079.stdout:302: left oid 31 (ObjNum 171 snap 27 seq_num 171) 2024-04-24T01:56:26.535 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:26.531+0000 7f6eb346d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:26.539 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:26.538+0000 7f6eb346d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:26.726 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:56:26.727 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-24T01:56:26.739 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":99,"stamp":"2024-04-24T01:56:26.017889+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":91196542,"num_objects":52,"num_object_clones":12,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":746,"num_read_kb":387764,"num_write":745,"num_write_kb":166700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":839,"ondisk_log_size":839,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":308160,"kb_used_data":200864,"kb_used_omap":0,"kb_used_meta":107264,"kb_avail":377179200,"statfs":{"total":386547056640,"available":386231500800,"internally_reserved":0,"allocated":205684736,"data_stored":219237753,"data_compressed":89889,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109838336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-8134456,"num_objects":1,"num_object_clones":2,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":36,"num_read_kb":25605,"num_write":34,"num_write_kb":8450,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.997735"},"pg_stats":[{"pgid":"3.f","version":"67'26","reported_seq":211,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-04-24T01:56:24.149820+0000","last_change":"2024-04-24T01:56:22.247006+0000","last_active":"2024-04-24T01:56:24.149820+0000","last_peered":"2024-04-24T01:56:24.149820+0000","last_clean":"2024-04-24T01:56:24.149820+0000","last_became_active":"2024-04-24T01:55:30.386726+0000","last_became_peered":"2024-04-24T01:55:30.386726+0000","last_unstale":"2024-04-24T01:56:24.149820+0000","last_undegraded":"2024-04-24T01:56:24.149820+0000","last_fullsized":"2024-04-24T01:56:24.149820+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0,"stat_sum":{"num_bytes":2437015,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":38,"num_read_kb":28311,"num_write":17,"num_write_kb":5216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"69'80","reported_seq":259,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-04-24T01:56:24.136941+0000","last_change":"2024-04-24T01:56:22.150144+0000","last_active":"2024-04-24T01:56:24.136941+0000","last_peered":"2024-04-24T01:56:24.136941+0000","last_clean":"2024-04-24T01:56:24.136941+0000","last_became_active":"2024-04-24T01:55:30.321866+0000","last_became_peered":"2024-04-24T01:55:30.321866+0000","last_unstale":"2024-04-24T01:56:24.136941+0000","last_undegraded":"2024-04-24T01:56:24.136941+0000","last_fullsized":"2024-04-24T01:56:24.136941+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0,"stat_sum":{"num_bytes":10218623,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":51,"num_read_kb":25527,"num_write":61,"num_write_kb":16505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"67'54","reported_seq":226,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-04-24T01:56:23.040045+0000","last_change":"2024-04-24T01:56:22.052276+0000","last_active":"2024-04-24T01:56:23.040045+0000","last_peered":"2024-04-24T01:56:23.040045+0000","last_clean":"2024-04-24T01:56:23.040045+0000","last_became_active":"2024-04-24T01:55:30.320186+0000","last_became_peered":"2024-04-24T01:55:30.320186+0000","last_unstale":"2024-04-24T01:56:23.040045+0000","last_undegraded":"2024-04-24T01:56:23.040045+0000","last_fullsized":"2024-04-24T01:56:23.040045+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":5,"snaptrim_duration":0,"stat_sum":{"num_bytes":5882417,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":26,"num_read_kb":8133,"num_write":48,"num_write_kb":12133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"67'65","reported_seq":249,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-04-24T01:56:24.145324+0000","last_change":"2024-04-24T01:56:22.246581+0000","last_active":"2024-04-24T01:56:24.145324+0000","last_peered":"2024-04-24T01:56:24.145324+0000","last_clean":"2024-04-24T01:56:24.145324+0000","last_became_active":"2024-04-24T01:55:30.288081+0000","last_became_peered":"2024-04-24T01:55:30.288081+0000","last_unstale":"2024-04-24T01:56:24.145324+0000","last_undegraded":"2024-04-24T01:56:24.145324+0000","last_fullsized":"2024-04-24T01:56:24.145324+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":8,"snaptrim_duration":0,"stat_sum":{"num_bytes":7395047,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":60,"num_read_kb":44504,"num_write":41,"num_write_kb":11889,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"39'23","reported_seq":207,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-04-24T01:56:24.047501+0000","last_change":"2024-04-24T01:56:22.258981+0000","last_active":"2024-04-24T01:56:24.047501+0000","last_peered":"2024-04-24T01:56:24.047501+0000","last_clean":"2024-04-24T01:56:24.047501+0000","last_became_active":"2024-04-24T01:55:30.333815+0000","last_became_peered":"2024-04-24T01:55:30.333815+0000","last_unstale":"2024-04-24T01:56:24.047501+0000","last_undegraded":"2024-04-24T01:56:24.047501+0000","last_fullsized":"2024-04-24T01:56:24.047501+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0,"stat_sum":{"num_bytes":3243718,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":28,"num_read_kb":17554,"num_write":25,"num_write_kb":6861,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"67'93","reported_seq":294,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-04-24T01:56:24.148538+0000","last_change":"2024-04-24T01:56:22.247349+0000","last_active":"2024-04-24T01:56:24.148538+0000","last_peered":"2024-04-24T01:56:24.148538+0000","last_clean":"2024-04-24T01:56:24.148538+0000","last_became_active":"2024-04-24T01:55:30.315075+0000","last_became_peered":"2024-04-24T01:55:30.315075+0000","last_unstale":"2024-04-24T01:56:24.148538+0000","last_undegraded":"2024-04-24T01:56:24.148538+0000","last_fullsized":"2024-04-24T01:56:24.148538+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":11,"snaptrim_duration":0,"stat_sum":{"num_bytes":14672445,"num_objects":7,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":90,"num_read_kb":52955,"num_write":83,"num_write_kb":22850,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"69'19","reported_seq":194,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-04-24T01:56:24.154081+0000","last_change":"2024-04-24T01:56:22.253377+0000","last_active":"2024-04-24T01:56:24.154081+0000","last_peered":"2024-04-24T01:56:24.154081+0000","last_clean":"2024-04-24T01:56:24.154081+0000","last_became_active":"2024-04-24T01:55:30.350300+0000","last_became_peered":"2024-04-24T01:55:30.350300+0000","last_unstale":"2024-04-24T01:56:24.154081+0000","last_undegraded":"2024-04-24T01:56:24.154081+0000","last_fullsized":"2024-04-24T01:56:24.154081+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":5003,"num_write":15,"num_write_kb":3526,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"64'6","reported_seq":174,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-04-24T01:56:23.048519+0000","last_change":"2024-04-24T01:56:22.055418+0000","last_active":"2024-04-24T01:56:23.048519+0000","last_peered":"2024-04-24T01:56:23.048519+0000","last_clean":"2024-04-24T01:56:23.048519+0000","last_became_active":"2024-04-24T01:55:30.376813+0000","last_became_peered":"2024-04-24T01:55:30.376813+0000","last_unstale":"2024-04-24T01:56:23.048519+0000","last_undegraded":"2024-04-24T01:56:23.048519+0000","last_fullsized":"2024-04-24T01:56:23.048519+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'97","reported_seq":290,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-04-24T01:56:24.039491+0000","last_change":"2024-04-24T01:55:16.196470+0000","last_active":"2024-04-24T01:56:24.039491+0000","last_peered":"2024-04-24T01:56:24.039491+0000","last_clean":"2024-04-24T01:56:24.039491+0000","last_became_active":"2024-04-24T01:55:16.190464+0000","last_became_peered":"2024-04-24T01:55:16.190464+0000","last_unstale":"2024-04-24T01:56:24.039491+0000","last_undegraded":"2024-04-24T01:56:24.039491+0000","last_fullsized":"2024-04-24T01:56:24.039491+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:15.127115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:15.127115+0000","last_clean_scrub_stamp":"2024-04-24T01:55:15.127115+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":154,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-04-24T01:56:23.048975+0000","last_change":"2024-04-24T01:55:18.236148+0000","last_active":"2024-04-24T01:56:23.048975+0000","last_peered":"2024-04-24T01:56:23.048975+0000","last_clean":"2024-04-24T01:56:23.048975+0000","last_became_active":"2024-04-24T01:55:18.230154+0000","last_became_peered":"2024-04-24T01:55:18.230154+0000","last_unstale":"2024-04-24T01:56:23.048975+0000","last_undegraded":"2024-04-24T01:56:23.048975+0000","last_fullsized":"2024-04-24T01:56:23.048975+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"68'66","reported_seq":260,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-04-24T01:56:23.049408+0000","last_change":"2024-04-24T01:56:22.054610+0000","last_active":"2024-04-24T01:56:23.049408+0000","last_peered":"2024-04-24T01:56:23.049408+0000","last_clean":"2024-04-24T01:56:23.049408+0000","last_became_active":"2024-04-24T01:55:30.316222+0000","last_became_peered":"2024-04-24T01:55:30.316222+0000","last_unstale":"2024-04-24T01:56:23.049408+0000","last_undegraded":"2024-04-24T01:56:23.049408+0000","last_fullsized":"2024-04-24T01:56:23.049408+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0,"stat_sum":{"num_bytes":4487469,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":74,"num_read_kb":46242,"num_write":56,"num_write_kb":15303,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":159,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-04-24T01:56:24.156239+0000","last_change":"2024-04-24T01:55:18.265905+0000","last_active":"2024-04-24T01:56:24.156239+0000","last_peered":"2024-04-24T01:56:24.156239+0000","last_clean":"2024-04-24T01:56:24.156239+0000","last_became_active":"2024-04-24T01:55:18.257020+0000","last_became_peered":"2024-04-24T01:55:18.257020+0000","last_unstale":"2024-04-24T01:56:24.156239+0000","last_undegraded":"2024-04-24T01:56:24.156239+0000","last_fullsized":"2024-04-24T01:56:24.156239+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"52'34","reported_seq":223,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-04-24T01:56:24.073359+0000","last_change":"2024-04-24T01:56:22.275985+0000","last_active":"2024-04-24T01:56:24.073359+0000","last_peered":"2024-04-24T01:56:24.073359+0000","last_clean":"2024-04-24T01:56:24.073359+0000","last_became_active":"2024-04-24T01:55:30.344129+0000","last_became_peered":"2024-04-24T01:55:30.344129+0000","last_unstale":"2024-04-24T01:56:24.073359+0000","last_undegraded":"2024-04-24T01:56:24.073359+0000","last_fullsized":"2024-04-24T01:56:24.073359+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0,"stat_sum":{"num_bytes":9109328,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":44,"num_read_kb":25529,"num_write":32,"num_write_kb":8371,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":154,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-04-24T01:56:23.032842+0000","last_change":"2024-04-24T01:55:18.215251+0000","last_active":"2024-04-24T01:56:23.032842+0000","last_peered":"2024-04-24T01:56:23.032842+0000","last_clean":"2024-04-24T01:56:23.032842+0000","last_became_active":"2024-04-24T01:55:18.206605+0000","last_became_peered":"2024-04-24T01:55:18.206605+0000","last_unstale":"2024-04-24T01:56:23.032842+0000","last_undegraded":"2024-04-24T01:56:23.032842+0000","last_fullsized":"2024-04-24T01:56:23.032842+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"39'22","reported_seq":199,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-04-24T01:56:24.045955+0000","last_change":"2024-04-24T01:56:22.260834+0000","last_active":"2024-04-24T01:56:24.045955+0000","last_peered":"2024-04-24T01:56:24.045955+0000","last_clean":"2024-04-24T01:56:24.045955+0000","last_became_active":"2024-04-24T01:55:30.401366+0000","last_became_peered":"2024-04-24T01:55:30.401366+0000","last_unstale":"2024-04-24T01:56:24.045955+0000","last_undegraded":"2024-04-24T01:56:24.045955+0000","last_fullsized":"2024-04-24T01:56:24.045955+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0,"stat_sum":{"num_bytes":7042067,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":9364,"num_write":18,"num_write_kb":5026,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":159,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-04-24T01:56:24.150672+0000","last_change":"2024-04-24T01:55:18.230893+0000","last_active":"2024-04-24T01:56:24.150672+0000","last_peered":"2024-04-24T01:56:24.150672+0000","last_clean":"2024-04-24T01:56:24.150672+0000","last_became_active":"2024-04-24T01:55:18.225188+0000","last_became_peered":"2024-04-24T01:55:18.225188+0000","last_unstale":"2024-04-24T01:56:24.150672+0000","last_undegraded":"2024-04-24T01:56:24.150672+0000","last_fullsized":"2024-04-24T01:56:24.150672+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"69'58","reported_seq":236,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-04-24T01:56:24.139134+0000","last_change":"2024-04-24T01:56:22.150445+0000","last_active":"2024-04-24T01:56:24.139134+0000","last_peered":"2024-04-24T01:56:24.139134+0000","last_clean":"2024-04-24T01:56:24.139134+0000","last_became_active":"2024-04-24T01:55:30.343318+0000","last_became_peered":"2024-04-24T01:55:30.343318+0000","last_unstale":"2024-04-24T01:56:24.139134+0000","last_undegraded":"2024-04-24T01:56:24.139134+0000","last_fullsized":"2024-04-24T01:56:24.139134+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":3,"snaptrim_duration":0,"stat_sum":{"num_bytes":6571504,"num_objects":6,"num_object_clones":3,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":41,"num_read_kb":28772,"num_write":46,"num_write_kb":13987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"68'75","reported_seq":262,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-04-24T01:56:23.080803+0000","last_change":"2024-04-24T01:56:22.097786+0000","last_active":"2024-04-24T01:56:23.080803+0000","last_peered":"2024-04-24T01:56:23.080803+0000","last_clean":"2024-04-24T01:56:23.080803+0000","last_became_active":"2024-04-24T01:55:30.345308+0000","last_became_peered":"2024-04-24T01:55:30.345308+0000","last_unstale":"2024-04-24T01:56:23.080803+0000","last_undegraded":"2024-04-24T01:56:23.080803+0000","last_fullsized":"2024-04-24T01:56:23.080803+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":7,"snaptrim_duration":0,"stat_sum":{"num_bytes":9495955,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":71,"num_read_kb":40506,"num_write":58,"num_write_kb":15036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":155,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-04-24T01:56:24.143486+0000","last_change":"2024-04-24T01:55:18.224238+0000","last_active":"2024-04-24T01:56:24.143486+0000","last_peered":"2024-04-24T01:56:24.143486+0000","last_clean":"2024-04-24T01:56:24.143486+0000","last_became_active":"2024-04-24T01:55:18.219131+0000","last_became_peered":"2024-04-24T01:55:18.219131+0000","last_unstale":"2024-04-24T01:56:24.143486+0000","last_undegraded":"2024-04-24T01:56:24.143486+0000","last_fullsized":"2024-04-24T01:56:24.143486+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"66'31","reported_seq":207,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-04-24T01:56:24.143805+0000","last_change":"2024-04-24T01:56:22.162185+0000","last_active":"2024-04-24T01:56:24.143805+0000","last_peered":"2024-04-24T01:56:24.143805+0000","last_clean":"2024-04-24T01:56:24.143805+0000","last_became_active":"2024-04-24T01:55:30.319958+0000","last_became_peered":"2024-04-24T01:55:30.319958+0000","last_unstale":"2024-04-24T01:56:24.143805+0000","last_undegraded":"2024-04-24T01:56:24.143805+0000","last_fullsized":"2024-04-24T01:56:24.143805+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0,"stat_sum":{"num_bytes":2638699,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":21,"num_read_kb":19606,"num_write":27,"num_write_kb":7720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":157,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-04-24T01:56:24.045824+0000","last_change":"2024-04-24T01:55:18.235763+0000","last_active":"2024-04-24T01:56:24.045824+0000","last_peered":"2024-04-24T01:56:24.045824+0000","last_clean":"2024-04-24T01:56:24.045824+0000","last_became_active":"2024-04-24T01:55:18.229175+0000","last_became_peered":"2024-04-24T01:55:18.229175+0000","last_unstale":"2024-04-24T01:56:24.045824+0000","last_undegraded":"2024-04-24T01:56:24.045824+0000","last_fullsized":"2024-04-24T01:56:24.045824+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"68'28","reported_seq":207,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-04-24T01:56:24.045669+0000","last_change":"2024-04-24T01:56:23.063637+0000","last_active":"2024-04-24T01:56:24.045669+0000","last_peered":"2024-04-24T01:56:24.045669+0000","last_clean":"2024-04-24T01:56:24.045669+0000","last_became_active":"2024-04-24T01:55:30.372690+0000","last_became_peered":"2024-04-24T01:55:30.372690+0000","last_unstale":"2024-04-24T01:56:24.045669+0000","last_undegraded":"2024-04-24T01:56:24.045669+0000","last_fullsized":"2024-04-24T01:56:24.045669+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":3,"snaptrim_duration":0,"stat_sum":{"num_bytes":2521050,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":28,"num_read_kb":16701,"num_write":24,"num_write_kb":6609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":155,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-04-24T01:56:24.146300+0000","last_change":"2024-04-24T01:55:18.278351+0000","last_active":"2024-04-24T01:56:24.146300+0000","last_peered":"2024-04-24T01:56:24.146300+0000","last_clean":"2024-04-24T01:56:24.146300+0000","last_became_active":"2024-04-24T01:55:18.273668+0000","last_became_peered":"2024-04-24T01:55:18.273668+0000","last_unstale":"2024-04-24T01:56:24.146300+0000","last_undegraded":"2024-04-24T01:56:24.146300+0000","last_fullsized":"2024-04-24T01:56:24.146300+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"66'62","reported_seq":239,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-04-24T01:56:24.144972+0000","last_change":"2024-04-24T01:56:22.162495+0000","last_active":"2024-04-24T01:56:24.144972+0000","last_peered":"2024-04-24T01:56:24.144972+0000","last_clean":"2024-04-24T01:56:24.144972+0000","last_became_active":"2024-04-24T01:55:30.363123+0000","last_became_peered":"2024-04-24T01:55:30.363123+0000","last_unstale":"2024-04-24T01:56:24.144972+0000","last_undegraded":"2024-04-24T01:56:24.144972+0000","last_fullsized":"2024-04-24T01:56:24.144972+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0,"stat_sum":{"num_bytes":4890837,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":39,"num_read_kb":18965,"num_write":52,"num_write_kb":12926,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":154,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-04-24T01:56:23.046914+0000","last_change":"2024-04-24T01:55:18.241117+0000","last_active":"2024-04-24T01:56:23.046914+0000","last_peered":"2024-04-24T01:56:23.046914+0000","last_clean":"2024-04-24T01:56:23.046914+0000","last_became_active":"2024-04-24T01:55:18.234313+0000","last_became_peered":"2024-04-24T01:55:18.234313+0000","last_unstale":"2024-04-24T01:56:23.046914+0000","last_undegraded":"2024-04-24T01:56:23.046914+0000","last_fullsized":"2024-04-24T01:56:23.046914+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":90606174,"num_objects":50,"num_object_clones":12,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":643,"num_read_kb":387673,"num_write":608,"num_write_kb":164806,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":205959168,"data_stored":222057788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":742,"ondisk_log_size":742,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":1}],"osd_stats":[{"osd":3,"up_from":10,"seq":42949673029,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":83820,"kb_used_data":56996,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94288020,"statfs":{"total":96636764160,"available":96550932480,"internally_reserved":0,"allocated":58363904,"data_stored":62196776,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673029,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":74720,"kb_used_data":47896,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94297120,"statfs":{"total":96636764160,"available":96560250880,"internally_reserved":0,"allocated":49045504,"data_stored":53847384,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705734,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":73836,"kb_used_data":47012,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94298004,"statfs":{"total":96636764160,"available":96561156096,"internally_reserved":0,"allocated":48140288,"data_stored":51675912,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705733,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":75784,"kb_used_data":48960,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94296056,"statfs":{"total":96636764160,"available":96559161344,"internally_reserved":0,"allocated":50135040,"data_stored":51517681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":48799744,"data_stored":52957136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":47288320,"data_stored":49019425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49631232,"data_stored":54815627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":60239872,"data_stored":65265600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T01:56:26.741 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-04-24T01:56:26.741 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-04-24T01:56:26.741 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T01:56:26.886 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:26.882+0000 7f3a6f8ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:26.890 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:26.889+0000 7f3a6f8ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:27.028 INFO:tasks.rados.rados.0.smithi079.stdout:298: done (5 left) 2024-04-24T01:56:27.028 INFO:tasks.rados.rados.0.smithi079.stdout:300: done (4 left) 2024-04-24T01:56:27.028 INFO:tasks.rados.rados.0.smithi079.stdout:302: done (3 left) 2024-04-24T01:56:27.028 INFO:tasks.rados.rados.0.smithi079.stdout:303: done (2 left) 2024-04-24T01:56:27.028 INFO:tasks.rados.rados.0.smithi079.stdout:304: done (1 left) 2024-04-24T01:56:27.028 INFO:tasks.rados.rados.0.smithi079.stdout:305: done (0 left) 2024-04-24T01:56:27.028 INFO:tasks.rados.rados.0.smithi079.stdout:306: read oid 16 snap -1 2024-04-24T01:56:27.028 INFO:tasks.rados.rados.0.smithi079.stdout:306: expect deleted 2024-04-24T01:56:27.028 INFO:tasks.rados.rados.0.smithi079.stdout:307: delete oid 37 current snap is 28 2024-04-24T01:56:27.083 INFO:tasks.rados.rados.0.smithi079.stdout:306: done (1 left) 2024-04-24T01:56:27.083 INFO:tasks.rados.rados.0.smithi079.stdout:307: done (0 left) 2024-04-24T01:56:27.083 INFO:tasks.rados.rados.0.smithi079.stdout:308: snap_remove snap 27 2024-04-24T01:56:27.091 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:56:27.092 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-24T01:56:27.104 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":101,"stamp":"2024-04-24T01:56:27.026279+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":91196542,"num_objects":52,"num_object_clones":12,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":746,"num_read_kb":387764,"num_write":745,"num_write_kb":166700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":839,"ondisk_log_size":839,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":308160,"kb_used_data":200864,"kb_used_omap":0,"kb_used_meta":107264,"kb_avail":377179200,"statfs":{"total":386547056640,"available":386231500800,"internally_reserved":0,"allocated":205684736,"data_stored":219237753,"data_compressed":89889,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109838336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-8134456,"num_objects":1,"num_object_clones":2,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":36,"num_read_kb":25605,"num_write":34,"num_write_kb":8450,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.005421"},"pg_stats":[{"pgid":"3.f","version":"67'26","reported_seq":211,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-04-24T01:56:24.149820+0000","last_change":"2024-04-24T01:56:22.247006+0000","last_active":"2024-04-24T01:56:24.149820+0000","last_peered":"2024-04-24T01:56:24.149820+0000","last_clean":"2024-04-24T01:56:24.149820+0000","last_became_active":"2024-04-24T01:55:30.386726+0000","last_became_peered":"2024-04-24T01:55:30.386726+0000","last_unstale":"2024-04-24T01:56:24.149820+0000","last_undegraded":"2024-04-24T01:56:24.149820+0000","last_fullsized":"2024-04-24T01:56:24.149820+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0,"stat_sum":{"num_bytes":2437015,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":38,"num_read_kb":28311,"num_write":17,"num_write_kb":5216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"69'80","reported_seq":259,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-04-24T01:56:24.136941+0000","last_change":"2024-04-24T01:56:22.150144+0000","last_active":"2024-04-24T01:56:24.136941+0000","last_peered":"2024-04-24T01:56:24.136941+0000","last_clean":"2024-04-24T01:56:24.136941+0000","last_became_active":"2024-04-24T01:55:30.321866+0000","last_became_peered":"2024-04-24T01:55:30.321866+0000","last_unstale":"2024-04-24T01:56:24.136941+0000","last_undegraded":"2024-04-24T01:56:24.136941+0000","last_fullsized":"2024-04-24T01:56:24.136941+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0,"stat_sum":{"num_bytes":10218623,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":51,"num_read_kb":25527,"num_write":61,"num_write_kb":16505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"67'54","reported_seq":226,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-04-24T01:56:23.040045+0000","last_change":"2024-04-24T01:56:22.052276+0000","last_active":"2024-04-24T01:56:23.040045+0000","last_peered":"2024-04-24T01:56:23.040045+0000","last_clean":"2024-04-24T01:56:23.040045+0000","last_became_active":"2024-04-24T01:55:30.320186+0000","last_became_peered":"2024-04-24T01:55:30.320186+0000","last_unstale":"2024-04-24T01:56:23.040045+0000","last_undegraded":"2024-04-24T01:56:23.040045+0000","last_fullsized":"2024-04-24T01:56:23.040045+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":5,"snaptrim_duration":0,"stat_sum":{"num_bytes":5882417,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":26,"num_read_kb":8133,"num_write":48,"num_write_kb":12133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"67'65","reported_seq":249,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-04-24T01:56:24.145324+0000","last_change":"2024-04-24T01:56:22.246581+0000","last_active":"2024-04-24T01:56:24.145324+0000","last_peered":"2024-04-24T01:56:24.145324+0000","last_clean":"2024-04-24T01:56:24.145324+0000","last_became_active":"2024-04-24T01:55:30.288081+0000","last_became_peered":"2024-04-24T01:55:30.288081+0000","last_unstale":"2024-04-24T01:56:24.145324+0000","last_undegraded":"2024-04-24T01:56:24.145324+0000","last_fullsized":"2024-04-24T01:56:24.145324+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":8,"snaptrim_duration":0,"stat_sum":{"num_bytes":7395047,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":60,"num_read_kb":44504,"num_write":41,"num_write_kb":11889,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"39'23","reported_seq":207,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-04-24T01:56:24.047501+0000","last_change":"2024-04-24T01:56:22.258981+0000","last_active":"2024-04-24T01:56:24.047501+0000","last_peered":"2024-04-24T01:56:24.047501+0000","last_clean":"2024-04-24T01:56:24.047501+0000","last_became_active":"2024-04-24T01:55:30.333815+0000","last_became_peered":"2024-04-24T01:55:30.333815+0000","last_unstale":"2024-04-24T01:56:24.047501+0000","last_undegraded":"2024-04-24T01:56:24.047501+0000","last_fullsized":"2024-04-24T01:56:24.047501+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0,"stat_sum":{"num_bytes":3243718,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":28,"num_read_kb":17554,"num_write":25,"num_write_kb":6861,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"67'93","reported_seq":294,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-04-24T01:56:24.148538+0000","last_change":"2024-04-24T01:56:22.247349+0000","last_active":"2024-04-24T01:56:24.148538+0000","last_peered":"2024-04-24T01:56:24.148538+0000","last_clean":"2024-04-24T01:56:24.148538+0000","last_became_active":"2024-04-24T01:55:30.315075+0000","last_became_peered":"2024-04-24T01:55:30.315075+0000","last_unstale":"2024-04-24T01:56:24.148538+0000","last_undegraded":"2024-04-24T01:56:24.148538+0000","last_fullsized":"2024-04-24T01:56:24.148538+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":11,"snaptrim_duration":0,"stat_sum":{"num_bytes":14672445,"num_objects":7,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":90,"num_read_kb":52955,"num_write":83,"num_write_kb":22850,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"69'19","reported_seq":194,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-04-24T01:56:24.154081+0000","last_change":"2024-04-24T01:56:22.253377+0000","last_active":"2024-04-24T01:56:24.154081+0000","last_peered":"2024-04-24T01:56:24.154081+0000","last_clean":"2024-04-24T01:56:24.154081+0000","last_became_active":"2024-04-24T01:55:30.350300+0000","last_became_peered":"2024-04-24T01:55:30.350300+0000","last_unstale":"2024-04-24T01:56:24.154081+0000","last_undegraded":"2024-04-24T01:56:24.154081+0000","last_fullsized":"2024-04-24T01:56:24.154081+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":5003,"num_write":15,"num_write_kb":3526,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"64'6","reported_seq":174,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-04-24T01:56:23.048519+0000","last_change":"2024-04-24T01:56:22.055418+0000","last_active":"2024-04-24T01:56:23.048519+0000","last_peered":"2024-04-24T01:56:23.048519+0000","last_clean":"2024-04-24T01:56:23.048519+0000","last_became_active":"2024-04-24T01:55:30.376813+0000","last_became_peered":"2024-04-24T01:55:30.376813+0000","last_unstale":"2024-04-24T01:56:23.048519+0000","last_undegraded":"2024-04-24T01:56:23.048519+0000","last_fullsized":"2024-04-24T01:56:23.048519+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'97","reported_seq":290,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-04-24T01:56:24.039491+0000","last_change":"2024-04-24T01:55:16.196470+0000","last_active":"2024-04-24T01:56:24.039491+0000","last_peered":"2024-04-24T01:56:24.039491+0000","last_clean":"2024-04-24T01:56:24.039491+0000","last_became_active":"2024-04-24T01:55:16.190464+0000","last_became_peered":"2024-04-24T01:55:16.190464+0000","last_unstale":"2024-04-24T01:56:24.039491+0000","last_undegraded":"2024-04-24T01:56:24.039491+0000","last_fullsized":"2024-04-24T01:56:24.039491+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:15.127115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:15.127115+0000","last_clean_scrub_stamp":"2024-04-24T01:55:15.127115+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":154,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-04-24T01:56:23.048975+0000","last_change":"2024-04-24T01:55:18.236148+0000","last_active":"2024-04-24T01:56:23.048975+0000","last_peered":"2024-04-24T01:56:23.048975+0000","last_clean":"2024-04-24T01:56:23.048975+0000","last_became_active":"2024-04-24T01:55:18.230154+0000","last_became_peered":"2024-04-24T01:55:18.230154+0000","last_unstale":"2024-04-24T01:56:23.048975+0000","last_undegraded":"2024-04-24T01:56:23.048975+0000","last_fullsized":"2024-04-24T01:56:23.048975+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"68'66","reported_seq":260,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-04-24T01:56:23.049408+0000","last_change":"2024-04-24T01:56:22.054610+0000","last_active":"2024-04-24T01:56:23.049408+0000","last_peered":"2024-04-24T01:56:23.049408+0000","last_clean":"2024-04-24T01:56:23.049408+0000","last_became_active":"2024-04-24T01:55:30.316222+0000","last_became_peered":"2024-04-24T01:55:30.316222+0000","last_unstale":"2024-04-24T01:56:23.049408+0000","last_undegraded":"2024-04-24T01:56:23.049408+0000","last_fullsized":"2024-04-24T01:56:23.049408+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0,"stat_sum":{"num_bytes":4487469,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":74,"num_read_kb":46242,"num_write":56,"num_write_kb":15303,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":159,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-04-24T01:56:24.156239+0000","last_change":"2024-04-24T01:55:18.265905+0000","last_active":"2024-04-24T01:56:24.156239+0000","last_peered":"2024-04-24T01:56:24.156239+0000","last_clean":"2024-04-24T01:56:24.156239+0000","last_became_active":"2024-04-24T01:55:18.257020+0000","last_became_peered":"2024-04-24T01:55:18.257020+0000","last_unstale":"2024-04-24T01:56:24.156239+0000","last_undegraded":"2024-04-24T01:56:24.156239+0000","last_fullsized":"2024-04-24T01:56:24.156239+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"52'34","reported_seq":223,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-04-24T01:56:24.073359+0000","last_change":"2024-04-24T01:56:22.275985+0000","last_active":"2024-04-24T01:56:24.073359+0000","last_peered":"2024-04-24T01:56:24.073359+0000","last_clean":"2024-04-24T01:56:24.073359+0000","last_became_active":"2024-04-24T01:55:30.344129+0000","last_became_peered":"2024-04-24T01:55:30.344129+0000","last_unstale":"2024-04-24T01:56:24.073359+0000","last_undegraded":"2024-04-24T01:56:24.073359+0000","last_fullsized":"2024-04-24T01:56:24.073359+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0,"stat_sum":{"num_bytes":9109328,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":44,"num_read_kb":25529,"num_write":32,"num_write_kb":8371,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":154,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-04-24T01:56:23.032842+0000","last_change":"2024-04-24T01:55:18.215251+0000","last_active":"2024-04-24T01:56:23.032842+0000","last_peered":"2024-04-24T01:56:23.032842+0000","last_clean":"2024-04-24T01:56:23.032842+0000","last_became_active":"2024-04-24T01:55:18.206605+0000","last_became_peered":"2024-04-24T01:55:18.206605+0000","last_unstale":"2024-04-24T01:56:23.032842+0000","last_undegraded":"2024-04-24T01:56:23.032842+0000","last_fullsized":"2024-04-24T01:56:23.032842+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"39'22","reported_seq":199,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-04-24T01:56:24.045955+0000","last_change":"2024-04-24T01:56:22.260834+0000","last_active":"2024-04-24T01:56:24.045955+0000","last_peered":"2024-04-24T01:56:24.045955+0000","last_clean":"2024-04-24T01:56:24.045955+0000","last_became_active":"2024-04-24T01:55:30.401366+0000","last_became_peered":"2024-04-24T01:55:30.401366+0000","last_unstale":"2024-04-24T01:56:24.045955+0000","last_undegraded":"2024-04-24T01:56:24.045955+0000","last_fullsized":"2024-04-24T01:56:24.045955+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0,"stat_sum":{"num_bytes":7042067,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":9364,"num_write":18,"num_write_kb":5026,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":159,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-04-24T01:56:24.150672+0000","last_change":"2024-04-24T01:55:18.230893+0000","last_active":"2024-04-24T01:56:24.150672+0000","last_peered":"2024-04-24T01:56:24.150672+0000","last_clean":"2024-04-24T01:56:24.150672+0000","last_became_active":"2024-04-24T01:55:18.225188+0000","last_became_peered":"2024-04-24T01:55:18.225188+0000","last_unstale":"2024-04-24T01:56:24.150672+0000","last_undegraded":"2024-04-24T01:56:24.150672+0000","last_fullsized":"2024-04-24T01:56:24.150672+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"69'58","reported_seq":236,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-04-24T01:56:24.139134+0000","last_change":"2024-04-24T01:56:22.150445+0000","last_active":"2024-04-24T01:56:24.139134+0000","last_peered":"2024-04-24T01:56:24.139134+0000","last_clean":"2024-04-24T01:56:24.139134+0000","last_became_active":"2024-04-24T01:55:30.343318+0000","last_became_peered":"2024-04-24T01:55:30.343318+0000","last_unstale":"2024-04-24T01:56:24.139134+0000","last_undegraded":"2024-04-24T01:56:24.139134+0000","last_fullsized":"2024-04-24T01:56:24.139134+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":3,"snaptrim_duration":0,"stat_sum":{"num_bytes":6571504,"num_objects":6,"num_object_clones":3,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":41,"num_read_kb":28772,"num_write":46,"num_write_kb":13987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"68'75","reported_seq":262,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-04-24T01:56:23.080803+0000","last_change":"2024-04-24T01:56:22.097786+0000","last_active":"2024-04-24T01:56:23.080803+0000","last_peered":"2024-04-24T01:56:23.080803+0000","last_clean":"2024-04-24T01:56:23.080803+0000","last_became_active":"2024-04-24T01:55:30.345308+0000","last_became_peered":"2024-04-24T01:55:30.345308+0000","last_unstale":"2024-04-24T01:56:23.080803+0000","last_undegraded":"2024-04-24T01:56:23.080803+0000","last_fullsized":"2024-04-24T01:56:23.080803+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":7,"snaptrim_duration":0,"stat_sum":{"num_bytes":9495955,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":71,"num_read_kb":40506,"num_write":58,"num_write_kb":15036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":155,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-04-24T01:56:24.143486+0000","last_change":"2024-04-24T01:55:18.224238+0000","last_active":"2024-04-24T01:56:24.143486+0000","last_peered":"2024-04-24T01:56:24.143486+0000","last_clean":"2024-04-24T01:56:24.143486+0000","last_became_active":"2024-04-24T01:55:18.219131+0000","last_became_peered":"2024-04-24T01:55:18.219131+0000","last_unstale":"2024-04-24T01:56:24.143486+0000","last_undegraded":"2024-04-24T01:56:24.143486+0000","last_fullsized":"2024-04-24T01:56:24.143486+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"66'31","reported_seq":207,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-04-24T01:56:24.143805+0000","last_change":"2024-04-24T01:56:22.162185+0000","last_active":"2024-04-24T01:56:24.143805+0000","last_peered":"2024-04-24T01:56:24.143805+0000","last_clean":"2024-04-24T01:56:24.143805+0000","last_became_active":"2024-04-24T01:55:30.319958+0000","last_became_peered":"2024-04-24T01:55:30.319958+0000","last_unstale":"2024-04-24T01:56:24.143805+0000","last_undegraded":"2024-04-24T01:56:24.143805+0000","last_fullsized":"2024-04-24T01:56:24.143805+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0,"stat_sum":{"num_bytes":2638699,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":21,"num_read_kb":19606,"num_write":27,"num_write_kb":7720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":157,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-04-24T01:56:24.045824+0000","last_change":"2024-04-24T01:55:18.235763+0000","last_active":"2024-04-24T01:56:24.045824+0000","last_peered":"2024-04-24T01:56:24.045824+0000","last_clean":"2024-04-24T01:56:24.045824+0000","last_became_active":"2024-04-24T01:55:18.229175+0000","last_became_peered":"2024-04-24T01:55:18.229175+0000","last_unstale":"2024-04-24T01:56:24.045824+0000","last_undegraded":"2024-04-24T01:56:24.045824+0000","last_fullsized":"2024-04-24T01:56:24.045824+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"68'28","reported_seq":207,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-04-24T01:56:24.045669+0000","last_change":"2024-04-24T01:56:23.063637+0000","last_active":"2024-04-24T01:56:24.045669+0000","last_peered":"2024-04-24T01:56:24.045669+0000","last_clean":"2024-04-24T01:56:24.045669+0000","last_became_active":"2024-04-24T01:55:30.372690+0000","last_became_peered":"2024-04-24T01:55:30.372690+0000","last_unstale":"2024-04-24T01:56:24.045669+0000","last_undegraded":"2024-04-24T01:56:24.045669+0000","last_fullsized":"2024-04-24T01:56:24.045669+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":3,"snaptrim_duration":0,"stat_sum":{"num_bytes":2521050,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":28,"num_read_kb":16701,"num_write":24,"num_write_kb":6609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":155,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-04-24T01:56:24.146300+0000","last_change":"2024-04-24T01:55:18.278351+0000","last_active":"2024-04-24T01:56:24.146300+0000","last_peered":"2024-04-24T01:56:24.146300+0000","last_clean":"2024-04-24T01:56:24.146300+0000","last_became_active":"2024-04-24T01:55:18.273668+0000","last_became_peered":"2024-04-24T01:55:18.273668+0000","last_unstale":"2024-04-24T01:56:24.146300+0000","last_undegraded":"2024-04-24T01:56:24.146300+0000","last_fullsized":"2024-04-24T01:56:24.146300+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"66'62","reported_seq":239,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-04-24T01:56:24.144972+0000","last_change":"2024-04-24T01:56:22.162495+0000","last_active":"2024-04-24T01:56:24.144972+0000","last_peered":"2024-04-24T01:56:24.144972+0000","last_clean":"2024-04-24T01:56:24.144972+0000","last_became_active":"2024-04-24T01:55:30.363123+0000","last_became_peered":"2024-04-24T01:55:30.363123+0000","last_unstale":"2024-04-24T01:56:24.144972+0000","last_undegraded":"2024-04-24T01:56:24.144972+0000","last_fullsized":"2024-04-24T01:56:24.144972+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0,"stat_sum":{"num_bytes":4890837,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":39,"num_read_kb":18965,"num_write":52,"num_write_kb":12926,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":154,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-04-24T01:56:23.046914+0000","last_change":"2024-04-24T01:55:18.241117+0000","last_active":"2024-04-24T01:56:23.046914+0000","last_peered":"2024-04-24T01:56:23.046914+0000","last_clean":"2024-04-24T01:56:23.046914+0000","last_became_active":"2024-04-24T01:55:18.234313+0000","last_became_peered":"2024-04-24T01:55:18.234313+0000","last_unstale":"2024-04-24T01:56:23.046914+0000","last_undegraded":"2024-04-24T01:56:23.046914+0000","last_fullsized":"2024-04-24T01:56:23.046914+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":90606174,"num_objects":50,"num_object_clones":12,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":643,"num_read_kb":387673,"num_write":608,"num_write_kb":164806,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":205959168,"data_stored":222057788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":742,"ondisk_log_size":742,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":1}],"osd_stats":[{"osd":3,"up_from":10,"seq":42949673029,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":83820,"kb_used_data":56996,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94288020,"statfs":{"total":96636764160,"available":96550932480,"internally_reserved":0,"allocated":58363904,"data_stored":62196776,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673029,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":74720,"kb_used_data":47896,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94297120,"statfs":{"total":96636764160,"available":96560250880,"internally_reserved":0,"allocated":49045504,"data_stored":53847384,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705734,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":73836,"kb_used_data":47012,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94298004,"statfs":{"total":96636764160,"available":96561156096,"internally_reserved":0,"allocated":48140288,"data_stored":51675912,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705733,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":75784,"kb_used_data":48960,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94296056,"statfs":{"total":96636764160,"available":96559161344,"internally_reserved":0,"allocated":50135040,"data_stored":51517681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":48799744,"data_stored":52957136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":47288320,"data_stored":49019425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49631232,"data_stored":54815627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":60239872,"data_stored":65265600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T01:56:28.030 INFO:tasks.rados.rados.0.smithi079.stdout:308: done (0 left) 2024-04-24T01:56:28.031 INFO:tasks.rados.rados.0.smithi079.stdout:309: snap_create 2024-04-24T01:56:28.519 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T01:56:28.634 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:28.631+0000 7fdad2e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:28.637 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:28.637+0000 7fdad2e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:29.041 INFO:tasks.rados.rados.0.smithi079.stdout:309: done (0 left) 2024-04-24T01:56:29.041 INFO:tasks.rados.rados.0.smithi079.stdout:310: write oid 32 current snap is 29 2024-04-24T01:56:29.041 INFO:tasks.rados.rados.0.smithi079.stdout:310: seq_num 174 ranges {524418=453092,1732455=588379,2779130=145288} 2024-04-24T01:56:29.046 INFO:tasks.rados.rados.0.smithi079.stdout:310: writing smithi07929762-32 from 524418 to 977510 tid 1 2024-04-24T01:56:29.056 INFO:tasks.rados.rados.0.smithi079.stdout:310: writing smithi07929762-32 from 1732455 to 2320834 tid 2 2024-04-24T01:56:29.061 INFO:tasks.rados.rados.0.smithi079.stdout:310: writing smithi07929762-32 from 2779130 to 2924418 tid 3 2024-04-24T01:56:29.061 INFO:tasks.rados.rados.0.smithi079.stdout:311: read oid 36 snap 28 2024-04-24T01:56:29.061 INFO:tasks.rados.rados.0.smithi079.stdout:311: expect deleted 2024-04-24T01:56:29.061 INFO:tasks.rados.rados.0.smithi079.stdout:312: snap_create 2024-04-24T01:56:29.125 INFO:tasks.rados.rados.0.smithi079.stdout:310: finishing write tid 1 to smithi07929762-32 2024-04-24T01:56:29.130 INFO:tasks.rados.rados.0.smithi079.stdout:310: finishing write tid 2 to smithi07929762-32 2024-04-24T01:56:29.132 INFO:tasks.rados.rados.0.smithi079.stdout:310: finishing write tid 3 to smithi07929762-32 2024-04-24T01:56:29.136 INFO:tasks.rados.rados.0.smithi079.stdout:310: finishing write tid 4 to smithi07929762-32 2024-04-24T01:56:29.138 INFO:tasks.rados.rados.0.smithi079.stdout:310: finishing write tid 5 to smithi07929762-32 2024-04-24T01:56:29.138 INFO:tasks.rados.rados.0.smithi079.stdout:310: oid 32 updating version 0 to 32 2024-04-24T01:56:29.139 INFO:tasks.rados.rados.0.smithi079.stdout:310: oid 32 version 32 is already newer than 32 2024-04-24T01:56:29.139 INFO:tasks.rados.rados.0.smithi079.stdout:310: oid 32 version 32 is already newer than 32 2024-04-24T01:56:29.139 INFO:tasks.rados.rados.0.smithi079.stdout:310: oid 32 version 32 is already newer than 32 2024-04-24T01:56:29.139 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 32 v 32 (ObjNum 174 snap 29 seq_num 174) dirty exists 2024-04-24T01:56:29.139 INFO:tasks.rados.rados.0.smithi079.stdout:310: left oid 32 (ObjNum 174 snap 29 seq_num 174) 2024-04-24T01:56:30.042 INFO:tasks.rados.rados.0.smithi079.stdout:310: done (2 left) 2024-04-24T01:56:30.042 INFO:tasks.rados.rados.0.smithi079.stdout:311: done (1 left) 2024-04-24T01:56:30.042 INFO:tasks.rados.rados.0.smithi079.stdout:312: done (0 left) 2024-04-24T01:56:30.043 INFO:tasks.rados.rados.0.smithi079.stdout:313: read oid 23 snap -1 2024-04-24T01:56:30.043 INFO:tasks.rados.rados.0.smithi079.stdout:313: expect deleted 2024-04-24T01:56:30.043 INFO:tasks.rados.rados.0.smithi079.stdout:314: snap_create 2024-04-24T01:56:30.734 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T01:56:30.848 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:30.846+0000 7fa9ad4ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:30.852 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:30.851+0000 7fa9ad4ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:31.046 INFO:tasks.rados.rados.0.smithi079.stdout:313: done (1 left) 2024-04-24T01:56:31.046 INFO:tasks.rados.rados.0.smithi079.stdout:314: done (0 left) 2024-04-24T01:56:31.047 INFO:tasks.rados.rados.0.smithi079.stdout:315: snap_remove snap 30 2024-04-24T01:56:32.047 INFO:tasks.rados.rados.0.smithi079.stdout:315: done (0 left) 2024-04-24T01:56:32.047 INFO:tasks.rados.rados.0.smithi079.stdout:316: snap_create 2024-04-24T01:56:32.106 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-04-24T01:56:32.106 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-04-24T01:56:32.106 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T01:56:32.218 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:32.214+0000 7fc96839d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:32.222 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:32.221+0000 7fc96839d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:32.407 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:56:32.407 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-24T01:56:32.419 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":108,"stamp":"2024-04-24T01:56:32.044486+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":93986570,"num_objects":43,"num_object_clones":5,"num_object_copies":129,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":757,"num_read_kb":394621,"num_write":766,"num_write_kb":171915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":885,"ondisk_log_size":885,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":284032,"kb_used_data":176736,"kb_used_omap":0,"kb_used_meta":107264,"kb_avail":377203328,"statfs":{"total":386547056640,"available":386256207872,"internally_reserved":0,"allocated":180977664,"data_stored":186637577,"data_compressed":89889,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109838336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2790028,"num_objects":-9,"num_object_clones":-7,"num_object_copies":-27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":-3,"num_read":11,"num_read_kb":6857,"num_write":21,"num_write_kb":5215,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.016903"},"pg_stats":[{"pgid":"3.f","version":"75'32","reported_seq":233,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-24T01:56:29.113124+0000","last_change":"2024-04-24T01:56:29.084022+0000","last_active":"2024-04-24T01:56:29.113124+0000","last_peered":"2024-04-24T01:56:29.113124+0000","last_clean":"2024-04-24T01:56:29.113124+0000","last_became_active":"2024-04-24T01:55:30.386726+0000","last_became_peered":"2024-04-24T01:55:30.386726+0000","last_unstale":"2024-04-24T01:56:29.113124+0000","last_undegraded":"2024-04-24T01:56:29.113124+0000","last_fullsized":"2024-04-24T01:56:29.113124+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0,"stat_sum":{"num_bytes":5361433,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":39,"num_read_kb":28312,"num_write":22,"num_write_kb":6376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"71'82","reported_seq":274,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-24T01:56:29.081532+0000","last_change":"2024-04-24T01:56:29.092553+0000","last_active":"2024-04-24T01:56:29.081532+0000","last_peered":"2024-04-24T01:56:29.081532+0000","last_clean":"2024-04-24T01:56:29.081532+0000","last_became_active":"2024-04-24T01:55:30.321866+0000","last_became_peered":"2024-04-24T01:55:30.321866+0000","last_unstale":"2024-04-24T01:56:29.081532+0000","last_undegraded":"2024-04-24T01:56:29.081532+0000","last_fullsized":"2024-04-24T01:56:29.081532+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0,"stat_sum":{"num_bytes":10218623,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":51,"num_read_kb":25527,"num_write":61,"num_write_kb":16505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"71'63","reported_seq":249,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-24T01:56:28.049517+0000","last_change":"2024-04-24T01:56:28.060918+0000","last_active":"2024-04-24T01:56:28.049517+0000","last_peered":"2024-04-24T01:56:28.049517+0000","last_clean":"2024-04-24T01:56:28.049517+0000","last_became_active":"2024-04-24T01:55:30.320186+0000","last_became_peered":"2024-04-24T01:55:30.320186+0000","last_unstale":"2024-04-24T01:56:28.049517+0000","last_undegraded":"2024-04-24T01:56:28.049517+0000","last_fullsized":"2024-04-24T01:56:28.049517+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":7,"snaptrim_duration":0,"stat_sum":{"num_bytes":5193363,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":27,"num_read_kb":8134,"num_write":53,"num_write_kb":13385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"71'67","reported_seq":266,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-24T01:56:29.076749+0000","last_change":"2024-04-24T01:56:29.083801+0000","last_active":"2024-04-24T01:56:29.076749+0000","last_peered":"2024-04-24T01:56:29.076749+0000","last_clean":"2024-04-24T01:56:29.076749+0000","last_became_active":"2024-04-24T01:55:30.288081+0000","last_became_peered":"2024-04-24T01:55:30.288081+0000","last_unstale":"2024-04-24T01:56:29.076749+0000","last_undegraded":"2024-04-24T01:56:29.076749+0000","last_fullsized":"2024-04-24T01:56:29.076749+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":9,"snaptrim_duration":0,"stat_sum":{"num_bytes":7395047,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":60,"num_read_kb":44504,"num_write":41,"num_write_kb":11889,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"39'23","reported_seq":223,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-24T01:56:29.096460+0000","last_change":"2024-04-24T01:56:28.243430+0000","last_active":"2024-04-24T01:56:29.096460+0000","last_peered":"2024-04-24T01:56:29.096460+0000","last_clean":"2024-04-24T01:56:29.096460+0000","last_became_active":"2024-04-24T01:55:30.333815+0000","last_became_peered":"2024-04-24T01:55:30.333815+0000","last_unstale":"2024-04-24T01:56:29.096460+0000","last_undegraded":"2024-04-24T01:56:29.096460+0000","last_fullsized":"2024-04-24T01:56:29.096460+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0,"stat_sum":{"num_bytes":3243718,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":28,"num_read_kb":17554,"num_write":25,"num_write_kb":6861,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"75'104","reported_seq":318,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-24T01:56:29.088021+0000","last_change":"2024-04-24T01:56:29.131685+0000","last_active":"2024-04-24T01:56:29.088021+0000","last_peered":"2024-04-24T01:56:29.088021+0000","last_clean":"2024-04-24T01:56:29.088021+0000","last_became_active":"2024-04-24T01:55:30.315075+0000","last_became_peered":"2024-04-24T01:55:30.315075+0000","last_unstale":"2024-04-24T01:56:29.088021+0000","last_undegraded":"2024-04-24T01:56:29.088021+0000","last_fullsized":"2024-04-24T01:56:29.088021+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":12,"snaptrim_duration":0,"stat_sum":{"num_bytes":12336305,"num_objects":8,"num_object_clones":2,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":91,"num_read_kb":52956,"num_write":89,"num_write_kb":24231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"69'19","reported_seq":210,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-24T01:56:29.086836+0000","last_change":"2024-04-24T01:56:29.089417+0000","last_active":"2024-04-24T01:56:29.086836+0000","last_peered":"2024-04-24T01:56:29.086836+0000","last_clean":"2024-04-24T01:56:29.086836+0000","last_became_active":"2024-04-24T01:55:30.350300+0000","last_became_peered":"2024-04-24T01:55:30.350300+0000","last_unstale":"2024-04-24T01:56:29.086836+0000","last_undegraded":"2024-04-24T01:56:29.086836+0000","last_fullsized":"2024-04-24T01:56:29.086836+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":5003,"num_write":15,"num_write_kb":3526,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"64'6","reported_seq":190,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-24T01:56:28.064478+0000","last_change":"2024-04-24T01:56:28.074432+0000","last_active":"2024-04-24T01:56:28.064478+0000","last_peered":"2024-04-24T01:56:28.064478+0000","last_clean":"2024-04-24T01:56:28.064478+0000","last_became_active":"2024-04-24T01:55:30.376813+0000","last_became_peered":"2024-04-24T01:55:30.376813+0000","last_unstale":"2024-04-24T01:56:28.064478+0000","last_undegraded":"2024-04-24T01:56:28.064478+0000","last_fullsized":"2024-04-24T01:56:28.064478+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'97","reported_seq":303,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-24T01:56:29.083964+0000","last_change":"2024-04-24T01:55:16.196470+0000","last_active":"2024-04-24T01:56:29.083964+0000","last_peered":"2024-04-24T01:56:29.083964+0000","last_clean":"2024-04-24T01:56:29.083964+0000","last_became_active":"2024-04-24T01:55:16.190464+0000","last_became_peered":"2024-04-24T01:55:16.190464+0000","last_unstale":"2024-04-24T01:56:29.083964+0000","last_undegraded":"2024-04-24T01:56:29.083964+0000","last_fullsized":"2024-04-24T01:56:29.083964+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:15.127115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:15.127115+0000","last_clean_scrub_stamp":"2024-04-24T01:55:15.127115+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":167,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-24T01:56:28.063344+0000","last_change":"2024-04-24T01:55:18.236148+0000","last_active":"2024-04-24T01:56:28.063344+0000","last_peered":"2024-04-24T01:56:28.063344+0000","last_clean":"2024-04-24T01:56:28.063344+0000","last_became_active":"2024-04-24T01:55:18.230154+0000","last_became_peered":"2024-04-24T01:55:18.230154+0000","last_unstale":"2024-04-24T01:56:28.063344+0000","last_undegraded":"2024-04-24T01:56:28.063344+0000","last_fullsized":"2024-04-24T01:56:28.063344+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"71'68","reported_seq":277,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-24T01:56:28.068164+0000","last_change":"2024-04-24T01:56:28.073672+0000","last_active":"2024-04-24T01:56:28.068164+0000","last_peered":"2024-04-24T01:56:28.068164+0000","last_clean":"2024-04-24T01:56:28.068164+0000","last_became_active":"2024-04-24T01:55:30.316222+0000","last_became_peered":"2024-04-24T01:55:30.316222+0000","last_unstale":"2024-04-24T01:56:28.068164+0000","last_undegraded":"2024-04-24T01:56:28.068164+0000","last_fullsized":"2024-04-24T01:56:28.068164+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":8,"snaptrim_duration":0,"stat_sum":{"num_bytes":4487469,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":74,"num_read_kb":46242,"num_write":56,"num_write_kb":15303,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":172,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-24T01:56:29.091239+0000","last_change":"2024-04-24T01:55:18.265905+0000","last_active":"2024-04-24T01:56:29.091239+0000","last_peered":"2024-04-24T01:56:29.091239+0000","last_clean":"2024-04-24T01:56:29.091239+0000","last_became_active":"2024-04-24T01:55:18.257020+0000","last_became_peered":"2024-04-24T01:55:18.257020+0000","last_unstale":"2024-04-24T01:56:29.091239+0000","last_undegraded":"2024-04-24T01:56:29.091239+0000","last_fullsized":"2024-04-24T01:56:29.091239+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"52'34","reported_seq":239,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-24T01:56:29.076426+0000","last_change":"2024-04-24T01:56:28.235058+0000","last_active":"2024-04-24T01:56:29.076426+0000","last_peered":"2024-04-24T01:56:29.076426+0000","last_clean":"2024-04-24T01:56:29.076426+0000","last_became_active":"2024-04-24T01:55:30.344129+0000","last_became_peered":"2024-04-24T01:55:30.344129+0000","last_unstale":"2024-04-24T01:56:29.076426+0000","last_undegraded":"2024-04-24T01:56:29.076426+0000","last_fullsized":"2024-04-24T01:56:29.076426+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0,"stat_sum":{"num_bytes":9109328,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":44,"num_read_kb":25529,"num_write":32,"num_write_kb":8371,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":167,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-24T01:56:28.050218+0000","last_change":"2024-04-24T01:55:18.215251+0000","last_active":"2024-04-24T01:56:28.050218+0000","last_peered":"2024-04-24T01:56:28.050218+0000","last_clean":"2024-04-24T01:56:28.050218+0000","last_became_active":"2024-04-24T01:55:18.206605+0000","last_became_peered":"2024-04-24T01:55:18.206605+0000","last_unstale":"2024-04-24T01:56:28.050218+0000","last_undegraded":"2024-04-24T01:56:28.050218+0000","last_fullsized":"2024-04-24T01:56:28.050218+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"39'22","reported_seq":215,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-24T01:56:29.089841+0000","last_change":"2024-04-24T01:56:28.241800+0000","last_active":"2024-04-24T01:56:29.089841+0000","last_peered":"2024-04-24T01:56:29.089841+0000","last_clean":"2024-04-24T01:56:29.089841+0000","last_became_active":"2024-04-24T01:55:30.401366+0000","last_became_peered":"2024-04-24T01:55:30.401366+0000","last_unstale":"2024-04-24T01:56:29.089841+0000","last_undegraded":"2024-04-24T01:56:29.089841+0000","last_fullsized":"2024-04-24T01:56:29.089841+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0,"stat_sum":{"num_bytes":7042067,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":9364,"num_write":18,"num_write_kb":5026,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":172,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-24T01:56:29.082291+0000","last_change":"2024-04-24T01:55:18.230893+0000","last_active":"2024-04-24T01:56:29.082291+0000","last_peered":"2024-04-24T01:56:29.082291+0000","last_clean":"2024-04-24T01:56:29.082291+0000","last_became_active":"2024-04-24T01:55:18.225188+0000","last_became_peered":"2024-04-24T01:55:18.225188+0000","last_unstale":"2024-04-24T01:56:29.082291+0000","last_undegraded":"2024-04-24T01:56:29.082291+0000","last_fullsized":"2024-04-24T01:56:29.082291+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"71'62","reported_seq":252,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-24T01:56:29.084378+0000","last_change":"2024-04-24T01:56:29.092243+0000","last_active":"2024-04-24T01:56:29.084378+0000","last_peered":"2024-04-24T01:56:29.084378+0000","last_clean":"2024-04-24T01:56:29.084378+0000","last_became_active":"2024-04-24T01:55:30.343318+0000","last_became_peered":"2024-04-24T01:55:30.343318+0000","last_unstale":"2024-04-24T01:56:29.084378+0000","last_undegraded":"2024-04-24T01:56:29.084378+0000","last_fullsized":"2024-04-24T01:56:29.084378+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0,"stat_sum":{"num_bytes":6571504,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":41,"num_read_kb":28772,"num_write":46,"num_write_kb":13987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"71'77","reported_seq":279,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-24T01:56:28.069335+0000","last_change":"2024-04-24T01:56:28.074653+0000","last_active":"2024-04-24T01:56:28.069335+0000","last_peered":"2024-04-24T01:56:28.069335+0000","last_clean":"2024-04-24T01:56:28.069335+0000","last_became_active":"2024-04-24T01:55:30.345308+0000","last_became_peered":"2024-04-24T01:55:30.345308+0000","last_unstale":"2024-04-24T01:56:28.069335+0000","last_undegraded":"2024-04-24T01:56:28.069335+0000","last_fullsized":"2024-04-24T01:56:28.069335+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":8,"snaptrim_duration":0,"stat_sum":{"num_bytes":9495955,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":71,"num_read_kb":40506,"num_write":58,"num_write_kb":15036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":166,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-24T01:56:29.094710+0000","last_change":"2024-04-24T01:55:18.224238+0000","last_active":"2024-04-24T01:56:29.094710+0000","last_peered":"2024-04-24T01:56:29.094710+0000","last_clean":"2024-04-24T01:56:29.094710+0000","last_became_active":"2024-04-24T01:55:18.219131+0000","last_became_peered":"2024-04-24T01:55:18.219131+0000","last_unstale":"2024-04-24T01:56:29.094710+0000","last_undegraded":"2024-04-24T01:56:29.094710+0000","last_fullsized":"2024-04-24T01:56:29.094710+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"71'33","reported_seq":222,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-24T01:56:29.096031+0000","last_change":"2024-04-24T01:56:29.105647+0000","last_active":"2024-04-24T01:56:29.096031+0000","last_peered":"2024-04-24T01:56:29.096031+0000","last_clean":"2024-04-24T01:56:29.096031+0000","last_became_active":"2024-04-24T01:55:30.319958+0000","last_became_peered":"2024-04-24T01:55:30.319958+0000","last_unstale":"2024-04-24T01:56:29.096031+0000","last_undegraded":"2024-04-24T01:56:29.096031+0000","last_fullsized":"2024-04-24T01:56:29.096031+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0,"stat_sum":{"num_bytes":2638699,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":21,"num_read_kb":19606,"num_write":27,"num_write_kb":7720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":170,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-24T01:56:29.095152+0000","last_change":"2024-04-24T01:55:18.235763+0000","last_active":"2024-04-24T01:56:29.095152+0000","last_peered":"2024-04-24T01:56:29.095152+0000","last_clean":"2024-04-24T01:56:29.095152+0000","last_became_active":"2024-04-24T01:55:18.229175+0000","last_became_peered":"2024-04-24T01:55:18.229175+0000","last_unstale":"2024-04-24T01:56:29.095152+0000","last_undegraded":"2024-04-24T01:56:29.095152+0000","last_fullsized":"2024-04-24T01:56:29.095152+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"68'28","reported_seq":224,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-24T01:56:29.084680+0000","last_change":"2024-04-24T01:56:28.236063+0000","last_active":"2024-04-24T01:56:29.084680+0000","last_peered":"2024-04-24T01:56:29.084680+0000","last_clean":"2024-04-24T01:56:29.084680+0000","last_became_active":"2024-04-24T01:55:30.372690+0000","last_became_peered":"2024-04-24T01:55:30.372690+0000","last_unstale":"2024-04-24T01:56:29.084680+0000","last_undegraded":"2024-04-24T01:56:29.084680+0000","last_fullsized":"2024-04-24T01:56:29.084680+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":3,"snaptrim_duration":0,"stat_sum":{"num_bytes":2521050,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":28,"num_read_kb":16701,"num_write":24,"num_write_kb":6609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":166,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-24T01:56:29.099612+0000","last_change":"2024-04-24T01:55:18.278351+0000","last_active":"2024-04-24T01:56:29.099612+0000","last_peered":"2024-04-24T01:56:29.099612+0000","last_clean":"2024-04-24T01:56:29.099612+0000","last_became_active":"2024-04-24T01:55:18.273668+0000","last_became_peered":"2024-04-24T01:55:18.273668+0000","last_unstale":"2024-04-24T01:56:29.099612+0000","last_undegraded":"2024-04-24T01:56:29.099612+0000","last_fullsized":"2024-04-24T01:56:29.099612+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"72'68","reported_seq":262,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-24T01:56:29.094762+0000","last_change":"2024-04-24T01:56:29.105006+0000","last_active":"2024-04-24T01:56:29.094762+0000","last_peered":"2024-04-24T01:56:29.094762+0000","last_clean":"2024-04-24T01:56:29.094762+0000","last_became_active":"2024-04-24T01:55:30.363123+0000","last_became_peered":"2024-04-24T01:55:30.363123+0000","last_unstale":"2024-04-24T01:56:29.094762+0000","last_undegraded":"2024-04-24T01:56:29.094762+0000","last_fullsized":"2024-04-24T01:56:29.094762+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":7,"snaptrim_duration":0,"stat_sum":{"num_bytes":7781641,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":47,"num_read_kb":25819,"num_write":57,"num_write_kb":14348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":167,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-24T01:56:28.065598+0000","last_change":"2024-04-24T01:55:18.241117+0000","last_active":"2024-04-24T01:56:28.065598+0000","last_peered":"2024-04-24T01:56:28.065598+0000","last_clean":"2024-04-24T01:56:28.065598+0000","last_became_active":"2024-04-24T01:55:18.234313+0000","last_became_peered":"2024-04-24T01:55:18.234313+0000","last_unstale":"2024-04-24T01:56:28.065598+0000","last_undegraded":"2024-04-24T01:56:28.065598+0000","last_fullsized":"2024-04-24T01:56:28.065598+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":93396202,"num_objects":41,"num_object_clones":5,"num_object_copies":123,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":654,"num_read_kb":394530,"num_write":629,"num_write_kb":170021,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":180801536,"data_stored":186276482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":788,"ondisk_log_size":788,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":1}],"osd_stats":[{"osd":3,"up_from":10,"seq":42949673034,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":76068,"kb_used_data":49244,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94295772,"statfs":{"total":96636764160,"available":96558870528,"internally_reserved":0,"allocated":50425856,"data_stored":52330478,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673034,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":70460,"kb_used_data":43636,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94301380,"statfs":{"total":96636764160,"available":96564613120,"internally_reserved":0,"allocated":44683264,"data_stored":46000360,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705739,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":65904,"kb_used_data":39080,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94305936,"statfs":{"total":96636764160,"available":96569278464,"internally_reserved":0,"allocated":40017920,"data_stored":41693292,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705738,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":71600,"kb_used_data":44776,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94300240,"statfs":{"total":96636764160,"available":96563445760,"internally_reserved":0,"allocated":45850624,"data_stored":46613447,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40550400,"data_stored":42046090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":45256704,"data_stored":46373475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":44027904,"data_stored":45170020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":50966528,"data_stored":52686897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T01:56:32.952 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T01:56:33.059 INFO:tasks.rados.rados.0.smithi079.stdout:316: done (0 left) 2024-04-24T01:56:33.059 INFO:tasks.rados.rados.0.smithi079.stdout:317: read oid 34 snap -1 2024-04-24T01:56:33.059 INFO:tasks.rados.rados.0.smithi079.stdout:317: expect (ObjNum 33 snap 0 seq_num 33) 2024-04-24T01:56:33.060 INFO:tasks.rados.rados.0.smithi079.stdout:318: delete oid 2 current snap is 32 2024-04-24T01:56:33.064 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:33.061+0000 7fc390f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:33.068 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:33.068+0000 7fc390f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:33.146 INFO:tasks.rados.rados.0.smithi079.stdout:317: expect (ObjNum 33 snap 0 seq_num 33) 2024-04-24T01:56:33.248 INFO:tasks.rados.rados.0.smithi079.stdout:317: done (1 left) 2024-04-24T01:56:33.248 INFO:tasks.rados.rados.0.smithi079.stdout:318: done (0 left) 2024-04-24T01:56:33.248 INFO:tasks.rados.rados.0.smithi079.stdout:319: snap_create 2024-04-24T01:56:34.059 INFO:tasks.rados.rados.0.smithi079.stdout:319: done (0 left) 2024-04-24T01:56:34.059 INFO:tasks.rados.rados.0.smithi079.stdout:320: read oid 2 snap -1 2024-04-24T01:56:34.059 INFO:tasks.rados.rados.0.smithi079.stdout:320: expect deleted 2024-04-24T01:56:34.059 INFO:tasks.rados.rados.0.smithi079.stdout:321: write (excl) oid 16 current snap is 33 2024-04-24T01:56:34.059 INFO:tasks.rados.rados.0.smithi079.stdout:321: seq_num 176 ranges {558032=719943,1849419=592048} 2024-04-24T01:56:34.066 INFO:tasks.rados.rados.0.smithi079.stdout:321: writing smithi07929762-16 from 558032 to 1277975 tid 1 2024-04-24T01:56:34.072 INFO:tasks.rados.rados.0.smithi079.stdout:321: writing smithi07929762-16 from 1849419 to 2441467 tid 2 2024-04-24T01:56:34.072 INFO:tasks.rados.rados.0.smithi079.stdout:322: write oid 13 current snap is 33 2024-04-24T01:56:34.072 INFO:tasks.rados.rados.0.smithi079.stdout:322: seq_num 177 ranges {574839=795192,1907901=452059} 2024-04-24T01:56:34.080 INFO:tasks.rados.rados.0.smithi079.stdout:322: writing smithi07929762-13 from 574839 to 1370031 tid 1 2024-04-24T01:56:34.086 INFO:tasks.rados.rados.0.smithi079.stdout:322: writing smithi07929762-13 from 1907901 to 2359960 tid 2 2024-04-24T01:56:34.086 INFO:tasks.rados.rados.0.smithi079.stdout:323: delete oid 22 current snap is 33 2024-04-24T01:56:34.138 INFO:tasks.rados.rados.0.smithi079.stdout:321: finishing write tid 1 to smithi07929762-16 2024-04-24T01:56:34.174 INFO:tasks.rados.rados.0.smithi079.stdout:320: done (3 left) 2024-04-24T01:56:34.175 INFO:tasks.rados.rados.0.smithi079.stdout:323: done (2 left) 2024-04-24T01:56:34.175 INFO:tasks.rados.rados.0.smithi079.stdout:324: delete oid 33 current snap is 33 2024-04-24T01:56:34.176 INFO:tasks.rados.rados.0.smithi079.stdout:322: finishing write tid 1 to smithi07929762-13 2024-04-24T01:56:34.176 INFO:tasks.rados.rados.0.smithi079.stdout:321: finishing write tid 2 to smithi07929762-16 2024-04-24T01:56:34.181 INFO:tasks.rados.rados.0.smithi079.stdout:322: finishing write tid 2 to smithi07929762-13 2024-04-24T01:56:34.185 INFO:tasks.rados.rados.0.smithi079.stdout:321: finishing write tid 3 to smithi07929762-16 2024-04-24T01:56:34.185 INFO:tasks.rados.rados.0.smithi079.stdout:322: finishing write tid 3 to smithi07929762-13 2024-04-24T01:56:34.186 INFO:tasks.rados.rados.0.smithi079.stdout:322: finishing write tid 4 to smithi07929762-13 2024-04-24T01:56:34.186 INFO:tasks.rados.rados.0.smithi079.stdout:322: oid 13 updating version 0 to 71 2024-04-24T01:56:34.186 INFO:tasks.rados.rados.0.smithi079.stdout:322: oid 13 version 71 is already newer than 71 2024-04-24T01:56:34.186 INFO:tasks.rados.rados.0.smithi079.stdout:322: oid 13 version 71 is already newer than 71 2024-04-24T01:56:34.186 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 13 v 71 (ObjNum 177 snap 33 seq_num 177) dirty exists 2024-04-24T01:56:34.186 INFO:tasks.rados.rados.0.smithi079.stdout:322: left oid 13 (ObjNum 177 snap 33 seq_num 177) 2024-04-24T01:56:34.186 INFO:tasks.rados.rados.0.smithi079.stdout:321: finishing write tid 4 to smithi07929762-16 2024-04-24T01:56:34.186 INFO:tasks.rados.rados.0.smithi079.stdout:321: oid 16 updating version 0 to 107 2024-04-24T01:56:34.187 INFO:tasks.rados.rados.0.smithi079.stdout:321: oid 16 version 107 is already newer than 107 2024-04-24T01:56:34.187 INFO:tasks.rados.rados.0.smithi079.stdout:321: oid 16 version 107 is already newer than 105 2024-04-24T01:56:34.187 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 16 v 107 (ObjNum 176 snap 33 seq_num 176) dirty exists 2024-04-24T01:56:34.187 INFO:tasks.rados.rados.0.smithi079.stdout:321: left oid 16 (ObjNum 176 snap 33 seq_num 176) 2024-04-24T01:56:34.205 INFO:tasks.rados.rados.0.smithi079.stdout:321: done (2 left) 2024-04-24T01:56:34.205 INFO:tasks.rados.rados.0.smithi079.stdout:322: done (1 left) 2024-04-24T01:56:34.205 INFO:tasks.rados.rados.0.smithi079.stdout:324: done (0 left) 2024-04-24T01:56:34.205 INFO:tasks.rados.rados.0.smithi079.stdout:325: delete oid 46 current snap is 33 2024-04-24T01:56:34.212 INFO:tasks.rados.rados.0.smithi079.stdout:325: done (0 left) 2024-04-24T01:56:34.212 INFO:tasks.rados.rados.0.smithi079.stdout:326: write oid 26 current snap is 33 2024-04-24T01:56:34.212 INFO:tasks.rados.rados.0.smithi079.stdout:326: seq_num 181 ranges {642067=696188,2025476=459397,2971091=70976} 2024-04-24T01:56:34.220 INFO:tasks.rados.rados.0.smithi079.stdout:326: writing smithi07929762-26 from 642067 to 1338255 tid 1 2024-04-24T01:56:34.227 INFO:tasks.rados.rados.0.smithi079.stdout:326: writing smithi07929762-26 from 2025476 to 2484873 tid 2 2024-04-24T01:56:34.230 INFO:tasks.rados.rados.0.smithi079.stdout:326: writing smithi07929762-26 from 2971091 to 3042067 tid 3 2024-04-24T01:56:34.230 INFO:tasks.rados.rados.0.smithi079.stdout:327: snap_remove snap 29 2024-04-24T01:56:35.072 INFO:tasks.rados.rados.0.smithi079.stdout:327: done (1 left) 2024-04-24T01:56:35.072 INFO:tasks.rados.rados.0.smithi079.stdout:328: snap_create 2024-04-24T01:56:35.072 INFO:tasks.rados.rados.0.smithi079.stdout:326: finishing write tid 1 to smithi07929762-26 2024-04-24T01:56:35.072 INFO:tasks.rados.rados.0.smithi079.stdout:326: finishing write tid 2 to smithi07929762-26 2024-04-24T01:56:35.072 INFO:tasks.rados.rados.0.smithi079.stdout:326: finishing write tid 3 to smithi07929762-26 2024-04-24T01:56:35.073 INFO:tasks.rados.rados.0.smithi079.stdout:326: finishing write tid 4 to smithi07929762-26 2024-04-24T01:56:35.073 INFO:tasks.rados.rados.0.smithi079.stdout:326: finishing write tid 5 to smithi07929762-26 2024-04-24T01:56:35.073 INFO:tasks.rados.rados.0.smithi079.stdout:326: oid 26 updating version 0 to 75 2024-04-24T01:56:35.073 INFO:tasks.rados.rados.0.smithi079.stdout:326: oid 26 version 75 is already newer than 74 2024-04-24T01:56:35.073 INFO:tasks.rados.rados.0.smithi079.stdout:326: oid 26 version 75 is already newer than 75 2024-04-24T01:56:35.073 INFO:tasks.rados.rados.0.smithi079.stdout:326: oid 26 version 75 is already newer than 75 2024-04-24T01:56:35.073 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 26 v 75 (ObjNum 181 snap 33 seq_num 181) dirty exists 2024-04-24T01:56:35.073 INFO:tasks.rados.rados.0.smithi079.stdout:326: left oid 26 (ObjNum 181 snap 33 seq_num 181) 2024-04-24T01:56:35.211 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T01:56:35.325 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:35.323+0000 7fdcf50ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:35.328 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:35.328+0000 7fdcf50ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:36.024 INFO:tasks.rados.rados.0.smithi079.stdout:326: done (1 left) 2024-04-24T01:56:36.025 INFO:tasks.rados.rados.0.smithi079.stdout:328: done (0 left) 2024-04-24T01:56:36.025 INFO:tasks.rados.rados.0.smithi079.stdout:329: write (excl) oid 29 current snap is 34 2024-04-24T01:56:36.025 INFO:tasks.rados.rados.0.smithi079.stdout:329: seq_num 182 ranges {658874=771437,2083958=603055} 2024-04-24T01:56:36.032 INFO:tasks.rados.rados.0.smithi079.stdout:329: writing smithi07929762-29 from 658874 to 1430311 tid 1 2024-04-24T01:56:36.040 INFO:tasks.rados.rados.0.smithi079.stdout:329: writing smithi07929762-29 from 2083958 to 2687013 tid 2 2024-04-24T01:56:36.040 INFO:tasks.rados.rados.0.smithi079.stdout:330: snap_remove snap 25 2024-04-24T01:56:37.028 INFO:tasks.rados.rados.0.smithi079.stdout:330: done (1 left) 2024-04-24T01:56:37.029 INFO:tasks.rados.rados.0.smithi079.stdout:331: write (excl) oid 28 current snap is 34 2024-04-24T01:56:37.029 INFO:tasks.rados.rados.0.smithi079.stdout:329: finishing write tid 1 to smithi07929762-29 2024-04-24T01:56:37.029 INFO:tasks.rados.rados.0.smithi079.stdout:331: seq_num 183 ranges {675681=563039,1858793=746713,3025937=49744} 2024-04-24T01:56:37.035 INFO:tasks.rados.rados.0.smithi079.stdout:331: writing smithi07929762-28 from 675681 to 1238720 tid 1 2024-04-24T01:56:37.042 INFO:tasks.rados.rados.0.smithi079.stdout:331: writing smithi07929762-28 from 1858793 to 2605506 tid 2 2024-04-24T01:56:37.045 INFO:tasks.rados.rados.0.smithi079.stdout:331: writing smithi07929762-28 from 3025937 to 3075681 tid 3 2024-04-24T01:56:37.045 INFO:tasks.rados.rados.0.smithi079.stdout:332: write oid 19 current snap is 34 2024-04-24T01:56:37.045 INFO:tasks.rados.rados.0.smithi079.stdout:332: seq_num 184 ranges {692488=638288,1917275=606724} 2024-04-24T01:56:37.052 INFO:tasks.rados.rados.0.smithi079.stdout:332: writing smithi07929762-19 from 692488 to 1330776 tid 1 2024-04-24T01:56:37.059 INFO:tasks.rados.rados.0.smithi079.stdout:332: writing smithi07929762-19 from 1917275 to 2523999 tid 2 2024-04-24T01:56:37.059 INFO:tasks.rados.rados.0.smithi079.stdout:333: read oid 41 snap 32 2024-04-24T01:56:37.059 INFO:tasks.rados.rados.0.smithi079.stdout:333: expect (ObjNum 113 snap 10 seq_num 113) 2024-04-24T01:56:37.059 INFO:tasks.rados.rados.0.smithi079.stdout:334: snap_remove snap 33 2024-04-24T01:56:37.421 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T01:56:37.422 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-04-24T01:56:37.422 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-04-24T01:56:37.422 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-04-24T01:56:37.422 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-04-24T01:56:37.528 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:37.525+0000 7f981f66d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:37.531 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:37.531+0000 7f981f66d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:37.532 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:37.526+0000 7f41ed66d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:37.535 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:37.535+0000 7f41ed66d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:37.553 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T01:56:37.553 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T01:56:37.553 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T01:56:37.553 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T01:56:37.554 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T01:56:37.554 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T01:56:37.554 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T01:56:37.554 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T01:56:37.554 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T01:56:37.554 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T01:56:37.554 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T01:56:37.554 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T01:56:37.556 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T01:56:37.557 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T01:56:37.557 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-04-24T01:56:37.667 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:37.661+0000 7febb4554640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:37.671 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:37.670+0000 7febb4554640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:37.685 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T01:56:37.685 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T01:56:37.685 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T01:56:37.685 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T01:56:37.685 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T01:56:37.685 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T01:56:37.685 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T01:56:37.685 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T01:56:37.686 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T01:56:37.686 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T01:56:37.686 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T01:56:37.686 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T01:56:37.688 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T01:56:37.689 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T01:56:37.689 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-04-24T01:56:37.836 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:37.831+0000 7f2ee486d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:37.840 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:37.839+0000 7f2ee486d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:37.856 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T01:56:37.856 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T01:56:37.856 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T01:56:37.856 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T01:56:37.856 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T01:56:37.856 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T01:56:37.857 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T01:56:37.857 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T01:56:37.857 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T01:56:37.857 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T01:56:37.857 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T01:56:37.857 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T01:56:37.859 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T01:56:37.859 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T01:56:37.859 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-04-24T01:56:37.968 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:37.962+0000 7f1fb566d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:37.972 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:37.971+0000 7f1fb566d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:37.989 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T01:56:37.989 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T01:56:37.989 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T01:56:37.989 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T01:56:37.989 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T01:56:37.989 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T01:56:37.989 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T01:56:37.989 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T01:56:37.990 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T01:56:37.990 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T01:56:37.990 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T01:56:37.990 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T01:56:37.991 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T01:56:37.991 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T01:56:38.030 INFO:tasks.rados.rados.0.smithi079.stdout:334: done (4 left) 2024-04-24T01:56:38.031 INFO:tasks.rados.rados.0.smithi079.stdout:335: write oid 18 current snap is 34 2024-04-24T01:56:38.031 INFO:tasks.rados.rados.0.smithi079.stdout:329: finishing write tid 2 to smithi07929762-29 2024-04-24T01:56:38.031 INFO:tasks.rados.rados.0.smithi079.stdout:335: seq_num 185 ranges {709295=713537,1859404=750382} 2024-04-24T01:56:38.038 INFO:tasks.rados.rados.0.smithi079.stdout:335: writing smithi07929762-18 from 709295 to 1422832 tid 1 2024-04-24T01:56:38.046 INFO:tasks.rados.rados.0.smithi079.stdout:335: writing smithi07929762-18 from 1859404 to 2609786 tid 2 2024-04-24T01:56:38.046 INFO:tasks.rados.rados.0.smithi079.stdout:336: read oid 47 snap -1 2024-04-24T01:56:38.046 INFO:tasks.rados.rados.0.smithi079.stdout:336: expect (ObjNum 130 snap 15 seq_num 130) 2024-04-24T01:56:38.046 INFO:tasks.rados.rados.0.smithi079.stdout:337: write (excl) oid 17 current snap is 34 2024-04-24T01:56:38.046 INFO:tasks.rados.rados.0.smithi079.stdout:337: seq_num 186 ranges {726102=788786,1917886=610393,3108206=17896} 2024-04-24T01:56:38.055 INFO:tasks.rados.rados.0.smithi079.stdout:337: writing smithi07929762-17 from 726102 to 1514888 tid 1 2024-04-24T01:56:38.063 INFO:tasks.rados.rados.0.smithi079.stdout:337: writing smithi07929762-17 from 1917886 to 2528279 tid 2 2024-04-24T01:56:38.066 INFO:tasks.rados.rados.0.smithi079.stdout:337: writing smithi07929762-17 from 3108206 to 3126102 tid 3 2024-04-24T01:56:38.066 INFO:tasks.rados.rados.0.smithi079.stdout:338: snap_remove snap 28 2024-04-24T01:56:39.034 INFO:tasks.rados.rados.0.smithi079.stdout:338: done (7 left) 2024-04-24T01:56:39.035 INFO:tasks.rados.rados.0.smithi079.stdout:339: read oid 45 snap -1 2024-04-24T01:56:39.035 INFO:tasks.rados.rados.0.smithi079.stdout:339: expect (ObjNum 152 snap 21 seq_num 152) 2024-04-24T01:56:39.035 INFO:tasks.rados.rados.0.smithi079.stdout:340: snap_create 2024-04-24T01:56:39.035 INFO:tasks.rados.rados.0.smithi079.stdout:329: finishing write tid 3 to smithi07929762-29 2024-04-24T01:56:39.035 INFO:tasks.rados.rados.0.smithi079.stdout:329: finishing write tid 4 to smithi07929762-29 2024-04-24T01:56:39.035 INFO:tasks.rados.rados.0.smithi079.stdout:329: oid 29 updating version 0 to 65 2024-04-24T01:56:39.035 INFO:tasks.rados.rados.0.smithi079.stdout:329: oid 29 updating version 65 to 67 2024-04-24T01:56:39.035 INFO:tasks.rados.rados.0.smithi079.stdout:329: oid 29 version 67 is already newer than 67 2024-04-24T01:56:39.035 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 29 v 67 (ObjNum 182 snap 34 seq_num 182) dirty exists 2024-04-24T01:56:39.035 INFO:tasks.rados.rados.0.smithi079.stdout:329: left oid 29 (ObjNum 182 snap 34 seq_num 182) 2024-04-24T01:56:39.035 INFO:tasks.rados.rados.0.smithi079.stdout:333: expect (ObjNum 113 snap 10 seq_num 113) 2024-04-24T01:56:39.241 INFO:tasks.rados.rados.0.smithi079.stdout:332: finishing write tid 1 to smithi07929762-19 2024-04-24T01:56:39.241 INFO:tasks.rados.rados.0.smithi079.stdout:332: finishing write tid 2 to smithi07929762-19 2024-04-24T01:56:39.242 INFO:tasks.rados.rados.0.smithi079.stdout:331: finishing write tid 1 to smithi07929762-28 2024-04-24T01:56:39.242 INFO:tasks.rados.rados.0.smithi079.stdout:332: finishing write tid 3 to smithi07929762-19 2024-04-24T01:56:39.242 INFO:tasks.rados.rados.0.smithi079.stdout:332: finishing write tid 4 to smithi07929762-19 2024-04-24T01:56:39.242 INFO:tasks.rados.rados.0.smithi079.stdout:332: oid 19 updating version 0 to 38 2024-04-24T01:56:39.242 INFO:tasks.rados.rados.0.smithi079.stdout:332: oid 19 version 38 is already newer than 38 2024-04-24T01:56:39.242 INFO:tasks.rados.rados.0.smithi079.stdout:332: oid 19 version 38 is already newer than 38 2024-04-24T01:56:39.242 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 19 v 38 (ObjNum 184 snap 34 seq_num 184) dirty exists 2024-04-24T01:56:39.242 INFO:tasks.rados.rados.0.smithi079.stdout:332: left oid 19 (ObjNum 184 snap 34 seq_num 184) 2024-04-24T01:56:39.242 INFO:tasks.rados.rados.0.smithi079.stdout:331: finishing write tid 2 to smithi07929762-28 2024-04-24T01:56:39.242 INFO:tasks.rados.rados.0.smithi079.stdout:331: finishing write tid 3 to smithi07929762-28 2024-04-24T01:56:39.242 INFO:tasks.rados.rados.0.smithi079.stdout:331: finishing write tid 4 to smithi07929762-28 2024-04-24T01:56:39.242 INFO:tasks.rados.rados.0.smithi079.stdout:331: finishing write tid 5 to smithi07929762-28 2024-04-24T01:56:39.242 INFO:tasks.rados.rados.0.smithi079.stdout:331: oid 28 updating version 0 to 89 2024-04-24T01:56:39.243 INFO:tasks.rados.rados.0.smithi079.stdout:331: oid 28 version 89 is already newer than 89 2024-04-24T01:56:39.243 INFO:tasks.rados.rados.0.smithi079.stdout:331: oid 28 version 89 is already newer than 89 2024-04-24T01:56:39.243 INFO:tasks.rados.rados.0.smithi079.stdout:331: oid 28 version 89 is already newer than 86 2024-04-24T01:56:39.243 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 28 v 89 (ObjNum 183 snap 34 seq_num 183) dirty exists 2024-04-24T01:56:39.243 INFO:tasks.rados.rados.0.smithi079.stdout:331: left oid 28 (ObjNum 183 snap 34 seq_num 183) 2024-04-24T01:56:39.243 INFO:tasks.rados.rados.0.smithi079.stdout:336: expect (ObjNum 130 snap 15 seq_num 130) 2024-04-24T01:56:39.340 INFO:tasks.rados.rados.0.smithi079.stdout:335: finishing write tid 1 to smithi07929762-18 2024-04-24T01:56:39.340 INFO:tasks.rados.rados.0.smithi079.stdout:337: finishing write tid 1 to smithi07929762-17 2024-04-24T01:56:39.341 INFO:tasks.rados.rados.0.smithi079.stdout:337: finishing write tid 2 to smithi07929762-17 2024-04-24T01:56:39.341 INFO:tasks.rados.rados.0.smithi079.stdout:337: finishing write tid 3 to smithi07929762-17 2024-04-24T01:56:39.341 INFO:tasks.rados.rados.0.smithi079.stdout:335: finishing write tid 2 to smithi07929762-18 2024-04-24T01:56:39.341 INFO:tasks.rados.rados.0.smithi079.stdout:335: finishing write tid 3 to smithi07929762-18 2024-04-24T01:56:39.341 INFO:tasks.rados.rados.0.smithi079.stdout:337: finishing write tid 4 to smithi07929762-17 2024-04-24T01:56:39.341 INFO:tasks.rados.rados.0.smithi079.stdout:335: finishing write tid 4 to smithi07929762-18 2024-04-24T01:56:39.341 INFO:tasks.rados.rados.0.smithi079.stdout:335: oid 18 updating version 0 to 115 2024-04-24T01:56:39.341 INFO:tasks.rados.rados.0.smithi079.stdout:335: oid 18 version 115 is already newer than 113 2024-04-24T01:56:39.341 INFO:tasks.rados.rados.0.smithi079.stdout:335: oid 18 version 115 is already newer than 115 2024-04-24T01:56:39.341 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 18 v 115 (ObjNum 185 snap 34 seq_num 185) dirty exists 2024-04-24T01:56:39.341 INFO:tasks.rados.rados.0.smithi079.stdout:335: left oid 18 (ObjNum 185 snap 34 seq_num 185) 2024-04-24T01:56:39.341 INFO:tasks.rados.rados.0.smithi079.stdout:337: finishing write tid 5 to smithi07929762-17 2024-04-24T01:56:39.341 INFO:tasks.rados.rados.0.smithi079.stdout:337: oid 17 updating version 0 to 73 2024-04-24T01:56:39.341 INFO:tasks.rados.rados.0.smithi079.stdout:337: oid 17 version 73 is already newer than 70 2024-04-24T01:56:39.342 INFO:tasks.rados.rados.0.smithi079.stdout:337: oid 17 version 73 is already newer than 73 2024-04-24T01:56:39.342 INFO:tasks.rados.rados.0.smithi079.stdout:337: oid 17 version 73 is already newer than 73 2024-04-24T01:56:39.342 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 17 v 73 (ObjNum 186 snap 34 seq_num 186) dirty exists 2024-04-24T01:56:39.342 INFO:tasks.rados.rados.0.smithi079.stdout:337: left oid 17 (ObjNum 186 snap 34 seq_num 186) 2024-04-24T01:56:39.342 INFO:tasks.rados.rados.0.smithi079.stdout:339: expect (ObjNum 152 snap 21 seq_num 152) 2024-04-24T01:56:39.628 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T01:56:39.733 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:39.731+0000 7fe87602d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:39.737 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:39.736+0000 7fe87602d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:40.035 INFO:tasks.rados.rados.0.smithi079.stdout:329: done (8 left) 2024-04-24T01:56:40.035 INFO:tasks.rados.rados.0.smithi079.stdout:331: done (7 left) 2024-04-24T01:56:40.035 INFO:tasks.rados.rados.0.smithi079.stdout:332: done (6 left) 2024-04-24T01:56:40.035 INFO:tasks.rados.rados.0.smithi079.stdout:333: done (5 left) 2024-04-24T01:56:40.036 INFO:tasks.rados.rados.0.smithi079.stdout:335: done (4 left) 2024-04-24T01:56:40.036 INFO:tasks.rados.rados.0.smithi079.stdout:336: done (3 left) 2024-04-24T01:56:40.036 INFO:tasks.rados.rados.0.smithi079.stdout:337: done (2 left) 2024-04-24T01:56:40.036 INFO:tasks.rados.rados.0.smithi079.stdout:339: done (1 left) 2024-04-24T01:56:40.036 INFO:tasks.rados.rados.0.smithi079.stdout:340: done (0 left) 2024-04-24T01:56:40.036 INFO:tasks.rados.rados.0.smithi079.stdout:341: snap_create 2024-04-24T01:56:41.042 INFO:tasks.rados.rados.0.smithi079.stdout:341: done (0 left) 2024-04-24T01:56:41.042 INFO:tasks.rados.rados.0.smithi079.stdout:342: read oid 24 snap 32 2024-04-24T01:56:41.043 INFO:tasks.rados.rados.0.smithi079.stdout:342: expect (ObjNum 56 snap 0 seq_num 56) 2024-04-24T01:56:41.043 INFO:tasks.rados.rados.0.smithi079.stdout:343: delete oid 27 current snap is 36 2024-04-24T01:56:41.148 INFO:tasks.rados.rados.0.smithi079.stdout:343: done (1 left) 2024-04-24T01:56:41.148 INFO:tasks.rados.rados.0.smithi079.stdout:344: write (excl) oid 35 current snap is 36 2024-04-24T01:56:41.148 INFO:tasks.rados.rados.0.smithi079.stdout:344: seq_num 188 ranges {759716=539284,2034850=614062} 2024-04-24T01:56:41.155 INFO:tasks.rados.rados.0.smithi079.stdout:344: writing smithi07929762-35 from 759716 to 1299000 tid 1 2024-04-24T01:56:41.162 INFO:tasks.rados.rados.0.smithi079.stdout:344: writing smithi07929762-35 from 2034850 to 2648912 tid 2 2024-04-24T01:56:41.162 INFO:tasks.rados.rados.0.smithi079.stdout:345: snap_create 2024-04-24T01:56:41.163 INFO:tasks.rados.rados.0.smithi079.stdout:342: expect (ObjNum 56 snap 0 seq_num 56) 2024-04-24T01:56:41.327 INFO:tasks.rados.rados.0.smithi079.stdout:344: finishing write tid 1 to smithi07929762-35 2024-04-24T01:56:41.327 INFO:tasks.rados.rados.0.smithi079.stdout:344: finishing write tid 2 to smithi07929762-35 2024-04-24T01:56:41.327 INFO:tasks.rados.rados.0.smithi079.stdout:344: finishing write tid 3 to smithi07929762-35 2024-04-24T01:56:41.327 INFO:tasks.rados.rados.0.smithi079.stdout:344: finishing write tid 4 to smithi07929762-35 2024-04-24T01:56:41.328 INFO:tasks.rados.rados.0.smithi079.stdout:344: oid 35 updating version 0 to 46 2024-04-24T01:56:41.328 INFO:tasks.rados.rados.0.smithi079.stdout:344: oid 35 version 46 is already newer than 46 2024-04-24T01:56:41.328 INFO:tasks.rados.rados.0.smithi079.stdout:344: oid 35 version 46 is already newer than 44 2024-04-24T01:56:41.328 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 35 v 46 (ObjNum 188 snap 36 seq_num 188) dirty exists 2024-04-24T01:56:41.328 INFO:tasks.rados.rados.0.smithi079.stdout:344: left oid 35 (ObjNum 188 snap 36 seq_num 188) 2024-04-24T01:56:41.831 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T01:56:41.938 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:41.936+0000 7fd67c86d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:41.942 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:41.941+0000 7fd67c86d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:42.045 INFO:tasks.rados.rados.0.smithi079.stdout:342: done (2 left) 2024-04-24T01:56:42.045 INFO:tasks.rados.rados.0.smithi079.stdout:344: done (1 left) 2024-04-24T01:56:42.045 INFO:tasks.rados.rados.0.smithi079.stdout:345: done (0 left) 2024-04-24T01:56:42.045 INFO:tasks.rados.rados.0.smithi079.stdout:346: write (excl) oid 13 current snap is 37 2024-04-24T01:56:42.045 INFO:tasks.rados.rados.0.smithi079.stdout:346: seq_num 189 ranges {776523=614533,1976979=757720} 2024-04-24T01:56:42.052 INFO:tasks.rados.rados.0.smithi079.stdout:346: writing smithi07929762-13 from 776523 to 1391056 tid 1 2024-04-24T01:56:42.060 INFO:tasks.rados.rados.0.smithi079.stdout:346: writing smithi07929762-13 from 1976979 to 2734699 tid 2 2024-04-24T01:56:42.060 INFO:tasks.rados.rados.0.smithi079.stdout:347: snap_create 2024-04-24T01:56:42.161 INFO:tasks.rados.rados.0.smithi079.stdout:346: finishing write tid 1 to smithi07929762-13 2024-04-24T01:56:42.191 INFO:tasks.rados.rados.0.smithi079.stdout:346: finishing write tid 2 to smithi07929762-13 2024-04-24T01:56:42.196 INFO:tasks.rados.rados.0.smithi079.stdout:346: finishing write tid 3 to smithi07929762-13 2024-04-24T01:56:42.197 INFO:tasks.rados.rados.0.smithi079.stdout:346: finishing write tid 4 to smithi07929762-13 2024-04-24T01:56:42.197 INFO:tasks.rados.rados.0.smithi079.stdout:346: oid 13 updating version 0 to 79 2024-04-24T01:56:42.197 INFO:tasks.rados.rados.0.smithi079.stdout:346: oid 13 version 79 is already newer than 77 2024-04-24T01:56:42.197 INFO:tasks.rados.rados.0.smithi079.stdout:346: oid 13 version 79 is already newer than 79 2024-04-24T01:56:42.197 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 13 v 79 (ObjNum 189 snap 37 seq_num 189) dirty exists 2024-04-24T01:56:42.197 INFO:tasks.rados.rados.0.smithi079.stdout:346: left oid 13 (ObjNum 189 snap 37 seq_num 189) 2024-04-24T01:56:42.993 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-04-24T01:56:42.993 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-04-24T01:56:42.993 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-04-24T01:56:42.993 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-04-24T01:56:43.056 INFO:tasks.rados.rados.0.smithi079.stdout:346: done (1 left) 2024-04-24T01:56:43.056 INFO:tasks.rados.rados.0.smithi079.stdout:347: done (0 left) 2024-04-24T01:56:43.056 INFO:tasks.rados.rados.0.smithi079.stdout:348: read oid 49 snap -1 2024-04-24T01:56:43.056 INFO:tasks.rados.rados.0.smithi079.stdout:348: expect (ObjNum 143 snap 18 seq_num 143) 2024-04-24T01:56:43.056 INFO:tasks.rados.rados.0.smithi079.stdout:349: read oid 33 snap 37 2024-04-24T01:56:43.056 INFO:tasks.rados.rados.0.smithi079.stdout:349: expect deleted 2024-04-24T01:56:43.057 INFO:tasks.rados.rados.0.smithi079.stdout:350: read oid 38 snap -1 2024-04-24T01:56:43.057 INFO:tasks.rados.rados.0.smithi079.stdout:350: expect deleted 2024-04-24T01:56:43.057 INFO:tasks.rados.rados.0.smithi079.stdout:351: snap_remove snap 35 2024-04-24T01:56:43.126 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:43.120+0000 7f46aaead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:43.129 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:43.129+0000 7f46aaead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:43.144 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T01:56:43.144 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T01:56:43.144 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T01:56:43.144 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T01:56:43.144 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T01:56:43.144 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T01:56:43.144 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T01:56:43.144 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T01:56:43.144 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T01:56:43.144 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T01:56:43.144 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T01:56:43.145 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T01:56:43.146 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T01:56:43.147 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T01:56:43.147 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-04-24T01:56:43.257 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:43.251+0000 7f626715d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:43.260 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:43.260+0000 7f626715d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:43.278 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T01:56:43.278 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T01:56:43.278 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T01:56:43.278 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T01:56:43.278 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T01:56:43.278 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T01:56:43.279 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T01:56:43.279 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T01:56:43.279 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T01:56:43.279 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T01:56:43.279 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T01:56:43.279 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T01:56:43.281 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T01:56:43.282 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T01:56:43.282 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-04-24T01:56:43.389 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:43.383+0000 7fd4cd86d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:43.393 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:43.392+0000 7fd4cd86d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:43.409 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T01:56:43.409 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T01:56:43.409 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T01:56:43.409 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T01:56:43.409 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T01:56:43.409 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T01:56:43.409 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T01:56:43.409 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T01:56:43.409 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T01:56:43.409 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T01:56:43.409 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T01:56:43.409 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T01:56:43.412 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T01:56:43.412 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T01:56:43.412 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-04-24T01:56:43.520 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:43.514+0000 7fc7ee22d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:43.524 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:43.523+0000 7fc7ee22d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:43.540 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T01:56:43.540 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T01:56:43.540 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T01:56:43.540 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T01:56:43.540 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T01:56:43.541 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T01:56:43.541 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T01:56:43.541 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T01:56:43.541 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T01:56:43.541 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T01:56:43.541 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T01:56:43.541 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T01:56:43.543 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T01:56:43.544 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T01:56:44.036 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T01:56:44.055 INFO:tasks.rados.rados.0.smithi079.stdout:351: done (3 left) 2024-04-24T01:56:44.055 INFO:tasks.rados.rados.0.smithi079.stdout:352: snap_remove snap 31 2024-04-24T01:56:44.152 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:44.149+0000 7f2f6206d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:44.155 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:44.154+0000 7f2f6206d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:45.055 INFO:tasks.rados.rados.0.smithi079.stdout:352: done (3 left) 2024-04-24T01:56:45.055 INFO:tasks.rados.rados.0.smithi079.stdout:353: snap_remove snap 36 2024-04-24T01:56:46.023 INFO:tasks.rados.rados.0.smithi079.stdout:353: done (3 left) 2024-04-24T01:56:46.024 INFO:tasks.rados.rados.0.smithi079.stdout:354: delete oid 15 current snap is 38 2024-04-24T01:56:46.024 INFO:tasks.rados.rados.0.smithi079.stdout:348: expect (ObjNum 143 snap 18 seq_num 143) 2024-04-24T01:56:46.132 INFO:tasks.rados.rados.0.smithi079.stdout:348: done (3 left) 2024-04-24T01:56:46.132 INFO:tasks.rados.rados.0.smithi079.stdout:349: done (2 left) 2024-04-24T01:56:46.132 INFO:tasks.rados.rados.0.smithi079.stdout:350: done (1 left) 2024-04-24T01:56:46.132 INFO:tasks.rados.rados.0.smithi079.stdout:354: done (0 left) 2024-04-24T01:56:46.132 INFO:tasks.rados.rados.0.smithi079.stdout:355: write (excl) oid 24 current snap is 38 2024-04-24T01:56:46.132 INFO:tasks.rados.rados.0.smithi079.stdout:355: seq_num 191 ranges {410137=481384,1410296=761389,2728968=481169} 2024-04-24T01:56:46.132 INFO:tasks.rados.rados.0.smithi079.stdout:355: writing smithi07929762-24 from 410137 to 891521 tid 1 2024-04-24T01:56:46.139 INFO:tasks.rados.rados.0.smithi079.stdout:355: writing smithi07929762-24 from 1410296 to 2171685 tid 2 2024-04-24T01:56:46.145 INFO:tasks.rados.rados.0.smithi079.stdout:355: writing smithi07929762-24 from 2728968 to 3210137 tid 3 2024-04-24T01:56:46.145 INFO:tasks.rados.rados.0.smithi079.stdout:356: read oid 7 snap -1 2024-04-24T01:56:46.145 INFO:tasks.rados.rados.0.smithi079.stdout:356: expect (ObjNum 167 snap 26 seq_num 167) 2024-04-24T01:56:46.145 INFO:tasks.rados.rados.0.smithi079.stdout:357: read oid 14 snap -1 2024-04-24T01:56:46.145 INFO:tasks.rados.rados.0.smithi079.stdout:357: expect (ObjNum 140 snap 17 seq_num 140) 2024-04-24T01:56:46.145 INFO:tasks.rados.rados.0.smithi079.stdout:358: write oid 36 current snap is 38 2024-04-24T01:56:46.146 INFO:tasks.rados.rados.0.smithi079.stdout:358: seq_num 192 ranges {426944=556633,1468778=621400,2872744=354200} 2024-04-24T01:56:46.150 INFO:tasks.rados.rados.0.smithi079.stdout:358: writing smithi07929762-36 from 426944 to 983577 tid 1 2024-04-24T01:56:46.156 INFO:tasks.rados.rados.0.smithi079.stdout:358: writing smithi07929762-36 from 1468778 to 2090178 tid 2 2024-04-24T01:56:46.163 INFO:tasks.rados.rados.0.smithi079.stdout:358: writing smithi07929762-36 from 2872744 to 3226944 tid 3 2024-04-24T01:56:46.164 INFO:tasks.rados.rados.0.smithi079.stdout:359: read oid 4 snap -1 2024-04-24T01:56:46.164 INFO:tasks.rados.rados.0.smithi079.stdout:359: expect deleted 2024-04-24T01:56:46.164 INFO:tasks.rados.rados.0.smithi079.stdout:360: read oid 45 snap -1 2024-04-24T01:56:46.164 INFO:tasks.rados.rados.0.smithi079.stdout:360: expect (ObjNum 152 snap 21 seq_num 152) 2024-04-24T01:56:46.164 INFO:tasks.rados.rados.0.smithi079.stdout:361: write oid 47 current snap is 38 2024-04-24T01:56:46.164 INFO:tasks.rados.rados.0.smithi079.stdout:361: seq_num 193 ranges {443751=631882,1810907=765058,3067461=176290} 2024-04-24T01:56:46.169 INFO:tasks.rados.rados.0.smithi079.stdout:361: writing smithi07929762-47 from 443751 to 1075633 tid 1 2024-04-24T01:56:46.179 INFO:tasks.rados.rados.0.smithi079.stdout:361: writing smithi07929762-47 from 1810907 to 2575965 tid 2 2024-04-24T01:56:46.182 INFO:tasks.rados.rados.0.smithi079.stdout:361: writing smithi07929762-47 from 3067461 to 3243751 tid 3 2024-04-24T01:56:46.183 INFO:tasks.rados.rados.0.smithi079.stdout:362: snap_create 2024-04-24T01:56:46.189 INFO:tasks.rados.rados.0.smithi079.stdout:356: expect (ObjNum 167 snap 26 seq_num 167) 2024-04-24T01:56:46.278 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T01:56:46.395 INFO:tasks.rados.rados.0.smithi079.stdout:360: expect (ObjNum 152 snap 21 seq_num 152) 2024-04-24T01:56:46.416 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:46.414+0000 7fed48aad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:46.420 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:46.419+0000 7fed48aad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:46.531 INFO:tasks.rados.rados.0.smithi079.stdout:358: finishing write tid 1 to smithi07929762-36 2024-04-24T01:56:46.531 INFO:tasks.rados.rados.0.smithi079.stdout:358: finishing write tid 2 to smithi07929762-36 2024-04-24T01:56:46.531 INFO:tasks.rados.rados.0.smithi079.stdout:358: finishing write tid 3 to smithi07929762-36 2024-04-24T01:56:46.531 INFO:tasks.rados.rados.0.smithi079.stdout:358: finishing write tid 4 to smithi07929762-36 2024-04-24T01:56:46.531 INFO:tasks.rados.rados.0.smithi079.stdout:358: finishing write tid 5 to smithi07929762-36 2024-04-24T01:56:46.532 INFO:tasks.rados.rados.0.smithi079.stdout:358: oid 36 updating version 0 to 33 2024-04-24T01:56:46.532 INFO:tasks.rados.rados.0.smithi079.stdout:358: oid 36 version 33 is already newer than 33 2024-04-24T01:56:46.532 INFO:tasks.rados.rados.0.smithi079.stdout:358: oid 36 version 33 is already newer than 33 2024-04-24T01:56:46.532 INFO:tasks.rados.rados.0.smithi079.stdout:358: oid 36 version 33 is already newer than 33 2024-04-24T01:56:46.532 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 36 v 33 (ObjNum 192 snap 38 seq_num 192) dirty exists 2024-04-24T01:56:46.532 INFO:tasks.rados.rados.0.smithi079.stdout:358: left oid 36 (ObjNum 192 snap 38 seq_num 192) 2024-04-24T01:56:46.532 INFO:tasks.rados.rados.0.smithi079.stdout:355: finishing write tid 1 to smithi07929762-24 2024-04-24T01:56:46.532 INFO:tasks.rados.rados.0.smithi079.stdout:357: expect (ObjNum 140 snap 17 seq_num 140) 2024-04-24T01:56:46.641 INFO:tasks.rados.rados.0.smithi079.stdout:361: finishing write tid 1 to smithi07929762-47 2024-04-24T01:56:46.641 INFO:tasks.rados.rados.0.smithi079.stdout:361: finishing write tid 2 to smithi07929762-47 2024-04-24T01:56:46.641 INFO:tasks.rados.rados.0.smithi079.stdout:355: finishing write tid 2 to smithi07929762-24 2024-04-24T01:56:46.642 INFO:tasks.rados.rados.0.smithi079.stdout:361: finishing write tid 3 to smithi07929762-47 2024-04-24T01:56:46.642 INFO:tasks.rados.rados.0.smithi079.stdout:361: finishing write tid 4 to smithi07929762-47 2024-04-24T01:56:46.642 INFO:tasks.rados.rados.0.smithi079.stdout:361: finishing write tid 5 to smithi07929762-47 2024-04-24T01:56:46.642 INFO:tasks.rados.rados.0.smithi079.stdout:361: oid 47 updating version 0 to 86 2024-04-24T01:56:46.642 INFO:tasks.rados.rados.0.smithi079.stdout:361: oid 47 version 86 is already newer than 86 2024-04-24T01:56:46.642 INFO:tasks.rados.rados.0.smithi079.stdout:361: oid 47 version 86 is already newer than 86 2024-04-24T01:56:46.642 INFO:tasks.rados.rados.0.smithi079.stdout:361: oid 47 version 86 is already newer than 86 2024-04-24T01:56:46.642 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 47 v 86 (ObjNum 193 snap 38 seq_num 193) dirty exists 2024-04-24T01:56:46.642 INFO:tasks.rados.rados.0.smithi079.stdout:361: left oid 47 (ObjNum 193 snap 38 seq_num 193) 2024-04-24T01:56:46.642 INFO:tasks.rados.rados.0.smithi079.stdout:355: finishing write tid 3 to smithi07929762-24 2024-04-24T01:56:46.642 INFO:tasks.rados.rados.0.smithi079.stdout:355: finishing write tid 4 to smithi07929762-24 2024-04-24T01:56:46.642 INFO:tasks.rados.rados.0.smithi079.stdout:355: finishing write tid 5 to smithi07929762-24 2024-04-24T01:56:46.643 INFO:tasks.rados.rados.0.smithi079.stdout:355: oid 24 updating version 0 to 57 2024-04-24T01:56:46.643 INFO:tasks.rados.rados.0.smithi079.stdout:355: oid 24 version 57 is already newer than 57 2024-04-24T01:56:46.643 INFO:tasks.rados.rados.0.smithi079.stdout:355: oid 24 version 57 is already newer than 57 2024-04-24T01:56:46.643 INFO:tasks.rados.rados.0.smithi079.stdout:355: oid 24 version 57 is already newer than 54 2024-04-24T01:56:46.643 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 24 v 57 (ObjNum 191 snap 38 seq_num 191) dirty exists 2024-04-24T01:56:46.643 INFO:tasks.rados.rados.0.smithi079.stdout:355: left oid 24 (ObjNum 191 snap 38 seq_num 191) 2024-04-24T01:56:47.027 INFO:tasks.rados.rados.0.smithi079.stdout:355: done (7 left) 2024-04-24T01:56:47.027 INFO:tasks.rados.rados.0.smithi079.stdout:356: done (6 left) 2024-04-24T01:56:47.027 INFO:tasks.rados.rados.0.smithi079.stdout:357: done (5 left) 2024-04-24T01:56:47.027 INFO:tasks.rados.rados.0.smithi079.stdout:358: done (4 left) 2024-04-24T01:56:47.027 INFO:tasks.rados.rados.0.smithi079.stdout:359: done (3 left) 2024-04-24T01:56:47.027 INFO:tasks.rados.rados.0.smithi079.stdout:360: done (2 left) 2024-04-24T01:56:47.027 INFO:tasks.rados.rados.0.smithi079.stdout:361: done (1 left) 2024-04-24T01:56:47.028 INFO:tasks.rados.rados.0.smithi079.stdout:362: done (0 left) 2024-04-24T01:56:47.028 INFO:tasks.rados.rados.0.smithi079.stdout:363: snap_remove snap 34 2024-04-24T01:56:48.030 INFO:tasks.rados.rados.0.smithi079.stdout:363: done (0 left) 2024-04-24T01:56:48.030 INFO:tasks.rados.rados.0.smithi079.stdout:364: snap_create 2024-04-24T01:56:48.511 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T01:56:48.544 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-04-24T01:56:48.545 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-04-24T01:56:48.619 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:48.616+0000 7fa3d60ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:48.622 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:48.622+0000 7fa3d60ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:48.668 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:48.661+0000 7f19f76ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:48.672 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:48.671+0000 7f19f76ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:49.031 INFO:teuthology.orchestra.run.smithi079.stderr:reweighted osd.0 to 1 (10000) 2024-04-24T01:56:49.034 INFO:tasks.rados.rados.0.smithi079.stdout:364: done (0 left) 2024-04-24T01:56:49.034 INFO:tasks.rados.rados.0.smithi079.stdout:365: write (excl) oid 4 current snap is 40 2024-04-24T01:56:49.034 INFO:tasks.rados.rados.0.smithi079.stdout:365: seq_num 194 ranges {460558=707131,1869389=508716,3094884=165674} 2024-04-24T01:56:49.040 INFO:tasks.rados.rados.0.smithi079.stdout:365: writing smithi07929762-4 from 460558 to 1167689 tid 1 2024-04-24T01:56:49.045 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-04-24T01:56:49.048 INFO:tasks.rados.rados.0.smithi079.stdout:365: writing smithi07929762-4 from 1869389 to 2378105 tid 2 2024-04-24T01:56:49.052 INFO:tasks.rados.rados.0.smithi079.stdout:365: writing smithi07929762-4 from 3094884 to 3260558 tid 3 2024-04-24T01:56:49.053 INFO:tasks.rados.rados.0.smithi079.stdout:366: write (excl) oid 22 current snap is 40 2024-04-24T01:56:49.053 INFO:tasks.rados.rados.0.smithi079.stdout:366: seq_num 195 ranges {477365=782380,1927871=768727,3122307=155058} 2024-04-24T01:56:49.060 INFO:tasks.rados.rados.0.smithi079.stdout:366: writing smithi07929762-22 from 477365 to 1259745 tid 1 2024-04-24T01:56:49.068 INFO:tasks.rados.rados.0.smithi079.stdout:366: writing smithi07929762-22 from 1927871 to 2696598 tid 2 2024-04-24T01:56:49.073 INFO:tasks.rados.rados.0.smithi079.stdout:366: writing smithi07929762-22 from 3122307 to 3277365 tid 3 2024-04-24T01:56:49.073 INFO:tasks.rados.rados.0.smithi079.stdout:367: write oid 34 current snap is 40 2024-04-24T01:56:49.073 INFO:tasks.rados.rados.0.smithi079.stdout:367: seq_num 196 ranges {494172=457629,1586353=512385,2749730=544442} 2024-04-24T01:56:49.078 INFO:tasks.rados.rados.0.smithi079.stdout:367: writing smithi07929762-34 from 494172 to 951801 tid 1 2024-04-24T01:56:49.085 INFO:tasks.rados.rados.0.smithi079.stdout:367: writing smithi07929762-34 from 1586353 to 2098738 tid 2 2024-04-24T01:56:49.094 INFO:tasks.rados.rados.0.smithi079.stdout:367: writing smithi07929762-34 from 2749730 to 3294172 tid 3 2024-04-24T01:56:49.094 INFO:tasks.rados.rados.0.smithi079.stdout:368: write oid 15 current snap is 40 2024-04-24T01:56:49.095 INFO:tasks.rados.rados.0.smithi079.stdout:368: seq_num 197 ranges {510979=532878,1528482=772396,3060800=250179} 2024-04-24T01:56:49.105 INFO:tasks.rados.rados.0.smithi079.stdout:368: writing smithi07929762-15 from 510979 to 1043857 tid 1 2024-04-24T01:56:49.114 INFO:tasks.rados.rados.0.smithi079.stdout:368: writing smithi07929762-15 from 1528482 to 2300878 tid 2 2024-04-24T01:56:49.120 INFO:tasks.rados.rados.0.smithi079.stdout:368: writing smithi07929762-15 from 3060800 to 3310979 tid 3 2024-04-24T01:56:49.121 INFO:tasks.rados.rados.0.smithi079.stdout:369: read oid 10 snap 38 2024-04-24T01:56:49.121 INFO:tasks.rados.rados.0.smithi079.stdout:369: expect (ObjNum 147 snap 20 seq_num 147) 2024-04-24T01:56:49.121 INFO:tasks.rados.rados.0.smithi079.stdout:370: write (excl) oid 12 current snap is 40 2024-04-24T01:56:49.121 INFO:tasks.rados.rados.0.smithi079.stdout:370: seq_num 198 ranges {527786=724480,1703317=516054,2804576=435035} 2024-04-24T01:56:49.127 INFO:tasks.rados.rados.0.smithi079.stdout:370: writing smithi07929762-12 from 527786 to 1252266 tid 1 2024-04-24T01:56:49.133 INFO:tasks.rados.rados.0.smithi079.stdout:370: writing smithi07929762-12 from 1703317 to 2219371 tid 2 2024-04-24T01:56:49.139 INFO:tasks.rados.rados.0.smithi079.stdout:370: writing smithi07929762-12 from 2804576 to 3239611 tid 3 2024-04-24T01:56:49.139 INFO:tasks.rados.rados.0.smithi079.stdout:371: read oid 37 snap 38 2024-04-24T01:56:49.139 INFO:tasks.rados.rados.0.smithi079.stdout:371: expect deleted 2024-04-24T01:56:49.139 INFO:tasks.rados.rados.0.smithi079.stdout:372: write (excl) oid 19 current snap is 40 2024-04-24T01:56:49.140 INFO:tasks.rados.rados.0.smithi079.stdout:366: finishing write tid 1 to smithi07929762-22 2024-04-24T01:56:49.140 INFO:tasks.rados.rados.0.smithi079.stdout:366: finishing write tid 2 to smithi07929762-22 2024-04-24T01:56:49.140 INFO:tasks.rados.rados.0.smithi079.stdout:372: seq_num 199 ranges {544593=799729,1761799=776065,2948352=396241} 2024-04-24T01:56:49.147 INFO:tasks.rados.rados.0.smithi079.stdout:372: writing smithi07929762-19 from 544593 to 1344322 tid 1 2024-04-24T01:56:49.155 INFO:tasks.rados.rados.0.smithi079.stdout:372: writing smithi07929762-19 from 1761799 to 2537864 tid 2 2024-04-24T01:56:49.161 INFO:tasks.rados.rados.0.smithi079.stdout:372: writing smithi07929762-19 from 2948352 to 3344593 tid 3 2024-04-24T01:56:49.161 INFO:tasks.rados.rados.0.smithi079.stdout:373: snap_create 2024-04-24T01:56:49.161 INFO:tasks.rados.rados.0.smithi079.stdout:366: finishing write tid 3 to smithi07929762-22 2024-04-24T01:56:49.161 INFO:tasks.rados.rados.0.smithi079.stdout:367: finishing write tid 1 to smithi07929762-34 2024-04-24T01:56:49.161 INFO:tasks.rados.rados.0.smithi079.stdout:365: finishing write tid 1 to smithi07929762-4 2024-04-24T01:56:49.161 INFO:tasks.rados.rados.0.smithi079.stdout:366: finishing write tid 4 to smithi07929762-22 2024-04-24T01:56:49.161 INFO:tasks.rados.rados.0.smithi079.stdout:366: finishing write tid 5 to smithi07929762-22 2024-04-24T01:56:49.161 INFO:tasks.rados.rados.0.smithi079.stdout:366: oid 22 updating version 0 to 36 2024-04-24T01:56:49.161 INFO:tasks.rados.rados.0.smithi079.stdout:366: oid 22 version 36 is already newer than 36 2024-04-24T01:56:49.161 INFO:tasks.rados.rados.0.smithi079.stdout:366: oid 22 version 36 is already newer than 33 2024-04-24T01:56:49.162 INFO:tasks.rados.rados.0.smithi079.stdout:366: oid 22 version 36 is already newer than 36 2024-04-24T01:56:49.162 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 22 v 36 (ObjNum 195 snap 40 seq_num 195) dirty exists 2024-04-24T01:56:49.162 INFO:tasks.rados.rados.0.smithi079.stdout:366: left oid 22 (ObjNum 195 snap 40 seq_num 195) 2024-04-24T01:56:49.168 INFO:tasks.rados.rados.0.smithi079.stdout:367: finishing write tid 2 to smithi07929762-34 2024-04-24T01:56:49.177 INFO:tasks.rados.rados.0.smithi079.stdout:367: finishing write tid 3 to smithi07929762-34 2024-04-24T01:56:49.177 INFO:tasks.rados.rados.0.smithi079.stdout:367: finishing write tid 4 to smithi07929762-34 2024-04-24T01:56:49.178 INFO:tasks.rados.rados.0.smithi079.stdout:367: finishing write tid 5 to smithi07929762-34 2024-04-24T01:56:49.178 INFO:tasks.rados.rados.0.smithi079.stdout:367: oid 34 updating version 0 to 73 2024-04-24T01:56:49.178 INFO:tasks.rados.rados.0.smithi079.stdout:367: oid 34 version 73 is already newer than 73 2024-04-24T01:56:49.178 INFO:tasks.rados.rados.0.smithi079.stdout:367: oid 34 version 73 is already newer than 73 2024-04-24T01:56:49.178 INFO:tasks.rados.rados.0.smithi079.stdout:367: oid 34 version 73 is already newer than 73 2024-04-24T01:56:49.178 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 34 v 73 (ObjNum 196 snap 40 seq_num 196) dirty exists 2024-04-24T01:56:49.178 INFO:tasks.rados.rados.0.smithi079.stdout:367: left oid 34 (ObjNum 196 snap 40 seq_num 196) 2024-04-24T01:56:49.191 INFO:tasks.rados.rados.0.smithi079.stdout:365: finishing write tid 2 to smithi07929762-4 2024-04-24T01:56:49.196 INFO:tasks.rados.rados.0.smithi079.stdout:369: expect (ObjNum 147 snap 20 seq_num 147) 2024-04-24T01:56:49.196 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:49.186+0000 7f172506d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:49.201 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:49.200+0000 7f172506d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:49.312 INFO:tasks.rados.rados.0.smithi079.stdout:365: finishing write tid 3 to smithi07929762-4 2024-04-24T01:56:49.312 INFO:tasks.rados.rados.0.smithi079.stdout:368: finishing write tid 1 to smithi07929762-15 2024-04-24T01:56:49.312 INFO:tasks.rados.rados.0.smithi079.stdout:368: finishing write tid 2 to smithi07929762-15 2024-04-24T01:56:49.312 INFO:tasks.rados.rados.0.smithi079.stdout:365: finishing write tid 4 to smithi07929762-4 2024-04-24T01:56:49.313 INFO:tasks.rados.rados.0.smithi079.stdout:365: finishing write tid 5 to smithi07929762-4 2024-04-24T01:56:49.313 INFO:tasks.rados.rados.0.smithi079.stdout:365: oid 4 updating version 0 to 20 2024-04-24T01:56:49.313 INFO:tasks.rados.rados.0.smithi079.stdout:365: oid 4 updating version 20 to 23 2024-04-24T01:56:49.313 INFO:tasks.rados.rados.0.smithi079.stdout:365: oid 4 version 23 is already newer than 23 2024-04-24T01:56:49.313 INFO:tasks.rados.rados.0.smithi079.stdout:365: oid 4 version 23 is already newer than 23 2024-04-24T01:56:49.313 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 4 v 23 (ObjNum 194 snap 40 seq_num 194) dirty exists 2024-04-24T01:56:49.313 INFO:tasks.rados.rados.0.smithi079.stdout:365: left oid 4 (ObjNum 194 snap 40 seq_num 194) 2024-04-24T01:56:49.313 INFO:tasks.rados.rados.0.smithi079.stdout:368: finishing write tid 3 to smithi07929762-15 2024-04-24T01:56:49.313 INFO:tasks.rados.rados.0.smithi079.stdout:372: finishing write tid 1 to smithi07929762-19 2024-04-24T01:56:49.313 INFO:tasks.rados.rados.0.smithi079.stdout:368: finishing write tid 4 to smithi07929762-15 2024-04-24T01:56:49.313 INFO:tasks.rados.rados.0.smithi079.stdout:368: finishing write tid 5 to smithi07929762-15 2024-04-24T01:56:49.313 INFO:tasks.rados.rados.0.smithi079.stdout:368: oid 15 updating version 0 to 85 2024-04-24T01:56:49.313 INFO:tasks.rados.rados.0.smithi079.stdout:368: oid 15 version 85 is already newer than 85 2024-04-24T01:56:49.313 INFO:tasks.rados.rados.0.smithi079.stdout:368: oid 15 version 85 is already newer than 85 2024-04-24T01:56:49.314 INFO:tasks.rados.rados.0.smithi079.stdout:368: oid 15 version 85 is already newer than 85 2024-04-24T01:56:49.314 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 15 v 85 (ObjNum 197 snap 40 seq_num 197) dirty exists 2024-04-24T01:56:49.314 INFO:tasks.rados.rados.0.smithi079.stdout:368: left oid 15 (ObjNum 197 snap 40 seq_num 197) 2024-04-24T01:56:49.314 INFO:tasks.rados.rados.0.smithi079.stdout:372: finishing write tid 2 to smithi07929762-19 2024-04-24T01:56:49.314 INFO:tasks.rados.rados.0.smithi079.stdout:370: finishing write tid 1 to smithi07929762-12 2024-04-24T01:56:49.314 INFO:tasks.rados.rados.0.smithi079.stdout:372: finishing write tid 3 to smithi07929762-19 2024-04-24T01:56:49.314 INFO:tasks.rados.rados.0.smithi079.stdout:372: finishing write tid 4 to smithi07929762-19 2024-04-24T01:56:49.314 INFO:tasks.rados.rados.0.smithi079.stdout:372: finishing write tid 5 to smithi07929762-19 2024-04-24T01:56:49.314 INFO:tasks.rados.rados.0.smithi079.stdout:372: oid 19 updating version 0 to 66 2024-04-24T01:56:49.314 INFO:tasks.rados.rados.0.smithi079.stdout:372: oid 19 version 66 is already newer than 63 2024-04-24T01:56:49.314 INFO:tasks.rados.rados.0.smithi079.stdout:372: oid 19 version 66 is already newer than 66 2024-04-24T01:56:49.314 INFO:tasks.rados.rados.0.smithi079.stdout:372: oid 19 version 66 is already newer than 66 2024-04-24T01:56:49.314 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 19 v 66 (ObjNum 199 snap 40 seq_num 199) dirty exists 2024-04-24T01:56:49.314 INFO:tasks.rados.rados.0.smithi079.stdout:372: left oid 19 (ObjNum 199 snap 40 seq_num 199) 2024-04-24T01:56:49.314 INFO:tasks.rados.rados.0.smithi079.stdout:370: finishing write tid 2 to smithi07929762-12 2024-04-24T01:56:49.315 INFO:tasks.rados.rados.0.smithi079.stdout:370: finishing write tid 3 to smithi07929762-12 2024-04-24T01:56:49.319 INFO:tasks.rados.rados.0.smithi079.stdout:370: finishing write tid 4 to smithi07929762-12 2024-04-24T01:56:49.321 INFO:tasks.rados.rados.0.smithi079.stdout:370: finishing write tid 5 to smithi07929762-12 2024-04-24T01:56:49.321 INFO:tasks.rados.rados.0.smithi079.stdout:370: oid 12 updating version 0 to 89 2024-04-24T01:56:49.321 INFO:tasks.rados.rados.0.smithi079.stdout:370: oid 12 version 89 is already newer than 86 2024-04-24T01:56:49.321 INFO:tasks.rados.rados.0.smithi079.stdout:370: oid 12 version 89 is already newer than 89 2024-04-24T01:56:49.321 INFO:tasks.rados.rados.0.smithi079.stdout:370: oid 12 version 89 is already newer than 89 2024-04-24T01:56:49.321 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 12 v 89 (ObjNum 198 snap 40 seq_num 198) dirty exists 2024-04-24T01:56:49.321 INFO:tasks.rados.rados.0.smithi079.stdout:370: left oid 12 (ObjNum 198 snap 40 seq_num 198) 2024-04-24T01:56:50.035 INFO:teuthology.orchestra.run.smithi079.stderr:reweighted osd.1 to 1 (10000) 2024-04-24T01:56:50.040 INFO:tasks.rados.rados.0.smithi079.stdout:365: done (8 left) 2024-04-24T01:56:50.040 INFO:tasks.rados.rados.0.smithi079.stdout:366: done (7 left) 2024-04-24T01:56:50.040 INFO:tasks.rados.rados.0.smithi079.stdout:367: done (6 left) 2024-04-24T01:56:50.040 INFO:tasks.rados.rados.0.smithi079.stdout:368: done (5 left) 2024-04-24T01:56:50.040 INFO:tasks.rados.rados.0.smithi079.stdout:369: done (4 left) 2024-04-24T01:56:50.041 INFO:tasks.rados.rados.0.smithi079.stdout:370: done (3 left) 2024-04-24T01:56:50.041 INFO:tasks.rados.rados.0.smithi079.stdout:371: done (2 left) 2024-04-24T01:56:50.041 INFO:tasks.rados.rados.0.smithi079.stdout:372: done (1 left) 2024-04-24T01:56:50.041 INFO:tasks.rados.rados.0.smithi079.stdout:373: done (0 left) 2024-04-24T01:56:50.041 INFO:tasks.rados.rados.0.smithi079.stdout:374: write oid 33 current snap is 41 2024-04-24T01:56:50.041 INFO:tasks.rados.rados.0.smithi079.stdout:374: seq_num 200 ranges {561400=474978,1820281=519723,2859422=457579} 2024-04-24T01:56:50.048 INFO:tasks.rados.rados.0.smithi079.stdout:374: writing smithi07929762-33 from 561400 to 1036378 tid 1 2024-04-24T01:56:50.055 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-04-24T01:56:50.056 INFO:tasks.rados.rados.0.smithi079.stdout:374: writing smithi07929762-33 from 1820281 to 2340004 tid 2 2024-04-24T01:56:50.064 INFO:tasks.rados.rados.0.smithi079.stdout:374: writing smithi07929762-33 from 2859422 to 3317001 tid 3 2024-04-24T01:56:50.064 INFO:tasks.rados.rados.0.smithi079.stdout:375: read oid 5 snap -1 2024-04-24T01:56:50.064 INFO:tasks.rados.rados.0.smithi079.stdout:375: expect (ObjNum 126 snap 15 seq_num 126) 2024-04-24T01:56:50.064 INFO:tasks.rados.rados.0.smithi079.stdout:376: read oid 32 snap -1 2024-04-24T01:56:50.064 INFO:tasks.rados.rados.0.smithi079.stdout:376: expect (ObjNum 174 snap 29 seq_num 174) 2024-04-24T01:56:50.064 INFO:tasks.rados.rados.0.smithi079.stdout:377: read oid 38 snap -1 2024-04-24T01:56:50.064 INFO:tasks.rados.rados.0.smithi079.stdout:377: expect deleted 2024-04-24T01:56:50.064 INFO:tasks.rados.rados.0.smithi079.stdout:378: snap_create 2024-04-24T01:56:50.126 INFO:tasks.rados.rados.0.smithi079.stdout:374: finishing write tid 1 to smithi07929762-33 2024-04-24T01:56:50.126 INFO:tasks.rados.rados.0.smithi079.stdout:374: finishing write tid 2 to smithi07929762-33 2024-04-24T01:56:50.134 INFO:tasks.rados.rados.0.smithi079.stdout:374: finishing write tid 3 to smithi07929762-33 2024-04-24T01:56:50.138 INFO:tasks.rados.rados.0.smithi079.stdout:374: finishing write tid 4 to smithi07929762-33 2024-04-24T01:56:50.139 INFO:tasks.rados.rados.0.smithi079.stdout:374: finishing write tid 5 to smithi07929762-33 2024-04-24T01:56:50.139 INFO:tasks.rados.rados.0.smithi079.stdout:374: oid 33 updating version 0 to 91 2024-04-24T01:56:50.139 INFO:tasks.rados.rados.0.smithi079.stdout:374: oid 33 version 91 is already newer than 91 2024-04-24T01:56:50.140 INFO:tasks.rados.rados.0.smithi079.stdout:374: oid 33 version 91 is already newer than 91 2024-04-24T01:56:50.140 INFO:tasks.rados.rados.0.smithi079.stdout:374: oid 33 version 91 is already newer than 91 2024-04-24T01:56:50.140 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 33 v 91 (ObjNum 200 snap 41 seq_num 200) dirty exists 2024-04-24T01:56:50.140 INFO:tasks.rados.rados.0.smithi079.stdout:374: left oid 33 (ObjNum 200 snap 41 seq_num 200) 2024-04-24T01:56:50.147 INFO:tasks.rados.rados.0.smithi079.stdout:375: expect (ObjNum 126 snap 15 seq_num 126) 2024-04-24T01:56:50.196 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:50.189+0000 7f0fbef5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:50.199 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:50.199+0000 7f0fbef5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:50.230 INFO:tasks.rados.rados.0.smithi079.stdout:376: expect (ObjNum 174 snap 29 seq_num 174) 2024-04-24T01:56:50.722 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T01:56:50.827 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:50.825+0000 7fd5fab54640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:50.831 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:50.830+0000 7fd5fab54640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:51.038 INFO:teuthology.orchestra.run.smithi079.stderr:reweighted osd.2 to 1 (10000) 2024-04-24T01:56:51.043 INFO:tasks.rados.rados.0.smithi079.stdout:374: done (4 left) 2024-04-24T01:56:51.043 INFO:tasks.rados.rados.0.smithi079.stdout:375: done (3 left) 2024-04-24T01:56:51.043 INFO:tasks.rados.rados.0.smithi079.stdout:376: done (2 left) 2024-04-24T01:56:51.043 INFO:tasks.rados.rados.0.smithi079.stdout:377: done (1 left) 2024-04-24T01:56:51.043 INFO:tasks.rados.rados.0.smithi079.stdout:378: done (0 left) 2024-04-24T01:56:51.043 INFO:tasks.rados.rados.0.smithi079.stdout:379: delete oid 8 current snap is 42 2024-04-24T01:56:51.059 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-04-24T01:56:51.079 INFO:tasks.rados.rados.0.smithi079.stdout:379: done (0 left) 2024-04-24T01:56:51.079 INFO:tasks.rados.rados.0.smithi079.stdout:380: read oid 14 snap -1 2024-04-24T01:56:51.080 INFO:tasks.rados.rados.0.smithi079.stdout:380: expect (ObjNum 140 snap 17 seq_num 140) 2024-04-24T01:56:51.080 INFO:tasks.rados.rados.0.smithi079.stdout:381: snap_create 2024-04-24T01:56:51.099 INFO:tasks.rados.rados.0.smithi079.stdout:380: expect (ObjNum 140 snap 17 seq_num 140) 2024-04-24T01:56:51.174 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:51.167+0000 7f49dfaad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:51.178 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:51.177+0000 7f49dfaad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:52.040 INFO:teuthology.orchestra.run.smithi079.stderr:reweighted osd.3 to 1 (10000) 2024-04-24T01:56:52.043 INFO:tasks.rados.rados.0.smithi079.stdout:380: done (1 left) 2024-04-24T01:56:52.043 INFO:tasks.rados.rados.0.smithi079.stdout:381: done (0 left) 2024-04-24T01:56:52.043 INFO:tasks.rados.rados.0.smithi079.stdout:382: snap_remove snap 40 2024-04-24T01:56:52.057 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-04-24T01:56:52.057 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T01:56:52.215 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:52.209+0000 7fe1d2ead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:52.219 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:52.219+0000 7fe1d2ead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:52.411 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:56:52.411 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-24T01:56:52.423 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":138,"stamp":"2024-04-24T01:56:52.041056+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":102432200,"num_objects":55,"num_object_clones":14,"num_object_copies":165,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":837,"num_read_kb":450649,"num_write":841,"num_write_kb":193846,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1037,"ondisk_log_size":1037,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":319320,"kb_used_data":206908,"kb_used_omap":0,"kb_used_meta":111424,"kb_avail":377168040,"statfs":{"total":386547056640,"available":386220072960,"internally_reserved":0,"allocated":211873792,"data_stored":222265341,"data_compressed":89889,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":114098176},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":8798445,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":-1,"num_read":25,"num_read_kb":18119,"num_write":28,"num_write_kb":9057,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.011846"},"pg_stats":[{"pgid":"3.f","version":"75'32","reported_seq":292,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-24T01:56:49.057838+0000","last_change":"2024-04-24T01:56:48.058716+0000","last_active":"2024-04-24T01:56:49.057838+0000","last_peered":"2024-04-24T01:56:49.057838+0000","last_clean":"2024-04-24T01:56:49.057838+0000","last_became_active":"2024-04-24T01:55:30.386726+0000","last_became_peered":"2024-04-24T01:55:30.386726+0000","last_unstale":"2024-04-24T01:56:49.057838+0000","last_undegraded":"2024-04-24T01:56:49.057838+0000","last_fullsized":"2024-04-24T01:56:49.057838+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0,"stat_sum":{"num_bytes":5361433,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":39,"num_read_kb":28312,"num_write":22,"num_write_kb":6376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"92'95","reported_seq":346,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-24T01:56:49.144135+0000","last_change":"2024-04-24T01:56:48.175633+0000","last_active":"2024-04-24T01:56:49.144135+0000","last_peered":"2024-04-24T01:56:49.144135+0000","last_clean":"2024-04-24T01:56:49.144135+0000","last_became_active":"2024-04-24T01:55:30.321866+0000","last_became_peered":"2024-04-24T01:55:30.321866+0000","last_unstale":"2024-04-24T01:56:49.144135+0000","last_undegraded":"2024-04-24T01:56:49.144135+0000","last_fullsized":"2024-04-24T01:56:49.144135+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":7,"snaptrim_duration":0,"stat_sum":{"num_bytes":10437114,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":65,"num_read_kb":37555,"num_write":66,"num_write_kb":17834,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"92'75","reported_seq":307,"reported_epoch":94,"state":"active+clean+snaptrim","last_fresh":"2024-04-24T01:56:48.150894+0000","last_change":"2024-04-24T01:56:46.170323+0000","last_active":"2024-04-24T01:56:48.150894+0000","last_peered":"2024-04-24T01:56:48.150894+0000","last_clean":"2024-04-24T01:56:48.150894+0000","last_became_active":"2024-04-24T01:55:30.320186+0000","last_became_peered":"2024-04-24T01:55:30.320186+0000","last_unstale":"2024-04-24T01:56:48.150894+0000","last_undegraded":"2024-04-24T01:56:48.150894+0000","last_fullsized":"2024-04-24T01:56:48.150894+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":8,"snaptrim_duration":0,"stat_sum":{"num_bytes":5411854,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":12901,"num_write":57,"num_write_kb":14728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"95'72","reported_seq":332,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-24T01:56:49.116649+0000","last_change":"2024-04-24T01:56:48.058499+0000","last_active":"2024-04-24T01:56:49.116649+0000","last_peered":"2024-04-24T01:56:49.116649+0000","last_clean":"2024-04-24T01:56:49.116649+0000","last_became_active":"2024-04-24T01:55:30.288081+0000","last_became_peered":"2024-04-24T01:55:30.288081+0000","last_unstale":"2024-04-24T01:56:49.116649+0000","last_undegraded":"2024-04-24T01:56:49.116649+0000","last_fullsized":"2024-04-24T01:56:49.116649+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":9,"snaptrim_duration":0,"stat_sum":{"num_bytes":8134588,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":66,"num_read_kb":49021,"num_write":46,"num_write_kb":13369,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"94'35","reported_seq":286,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-24T01:56:49.062621+0000","last_change":"2024-04-24T01:56:48.251963+0000","last_active":"2024-04-24T01:56:49.062621+0000","last_peered":"2024-04-24T01:56:49.062621+0000","last_clean":"2024-04-24T01:56:49.062621+0000","last_became_active":"2024-04-24T01:55:30.333815+0000","last_became_peered":"2024-04-24T01:55:30.333815+0000","last_unstale":"2024-04-24T01:56:49.062621+0000","last_undegraded":"2024-04-24T01:56:49.062621+0000","last_fullsized":"2024-04-24T01:56:49.062621+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0,"stat_sum":{"num_bytes":3226944,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":30,"num_read_kb":17555,"num_write":31,"num_write_kb":8358,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"91'119","reported_seq":392,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-24T01:56:49.062340+0000","last_change":"2024-04-24T01:56:48.058276+0000","last_active":"2024-04-24T01:56:49.062340+0000","last_peered":"2024-04-24T01:56:49.062340+0000","last_clean":"2024-04-24T01:56:49.062340+0000","last_became_active":"2024-04-24T01:55:30.315075+0000","last_became_peered":"2024-04-24T01:55:30.315075+0000","last_unstale":"2024-04-24T01:56:49.062340+0000","last_undegraded":"2024-04-24T01:56:49.062340+0000","last_fullsized":"2024-04-24T01:56:49.062340+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":15,"snaptrim_duration":0,"stat_sum":{"num_bytes":16117913,"num_objects":7,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":100,"num_read_kb":58593,"num_write":97,"num_write_kb":26944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"95'20","reported_seq":271,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-24T01:56:49.101772+0000","last_change":"2024-04-24T01:56:48.063986+0000","last_active":"2024-04-24T01:56:49.101772+0000","last_peered":"2024-04-24T01:56:49.101772+0000","last_clean":"2024-04-24T01:56:49.101772+0000","last_became_active":"2024-04-24T01:55:30.350300+0000","last_became_peered":"2024-04-24T01:55:30.350300+0000","last_unstale":"2024-04-24T01:56:49.101772+0000","last_undegraded":"2024-04-24T01:56:49.101772+0000","last_fullsized":"2024-04-24T01:56:49.101772+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0,"stat_sum":{"num_bytes":2378105,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":13,"num_read_kb":5003,"num_write":17,"num_write_kb":4714,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"64'6","reported_seq":237,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-24T01:56:46.272687+0000","last_change":"2024-04-24T01:56:46.272687+0000","last_active":"2024-04-24T01:56:46.272687+0000","last_peered":"2024-04-24T01:56:46.272687+0000","last_clean":"2024-04-24T01:56:46.272687+0000","last_became_active":"2024-04-24T01:55:30.376813+0000","last_became_peered":"2024-04-24T01:55:30.376813+0000","last_unstale":"2024-04-24T01:56:46.272687+0000","last_undegraded":"2024-04-24T01:56:46.272687+0000","last_fullsized":"2024-04-24T01:56:46.272687+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'97","reported_seq":349,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-24T01:56:49.047435+0000","last_change":"2024-04-24T01:55:16.196470+0000","last_active":"2024-04-24T01:56:49.047435+0000","last_peered":"2024-04-24T01:56:49.047435+0000","last_clean":"2024-04-24T01:56:49.047435+0000","last_became_active":"2024-04-24T01:55:16.190464+0000","last_became_peered":"2024-04-24T01:55:16.190464+0000","last_unstale":"2024-04-24T01:56:49.047435+0000","last_undegraded":"2024-04-24T01:56:49.047435+0000","last_fullsized":"2024-04-24T01:56:49.047435+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:15.127115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:15.127115+0000","last_clean_scrub_stamp":"2024-04-24T01:55:15.127115+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":206,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-24T01:56:46.188874+0000","last_change":"2024-04-24T01:55:18.236148+0000","last_active":"2024-04-24T01:56:46.188874+0000","last_peered":"2024-04-24T01:56:46.188874+0000","last_clean":"2024-04-24T01:56:46.188874+0000","last_became_active":"2024-04-24T01:55:18.230154+0000","last_became_peered":"2024-04-24T01:55:18.230154+0000","last_unstale":"2024-04-24T01:56:46.188874+0000","last_undegraded":"2024-04-24T01:56:46.188874+0000","last_fullsized":"2024-04-24T01:56:46.188874+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"92'86","reported_seq":344,"reported_epoch":94,"state":"active+clean+snaptrim","last_fresh":"2024-04-24T01:56:46.378284+0000","last_change":"2024-04-24T01:56:46.197594+0000","last_active":"2024-04-24T01:56:46.378284+0000","last_peered":"2024-04-24T01:56:46.378284+0000","last_clean":"2024-04-24T01:56:46.378284+0000","last_became_active":"2024-04-24T01:55:30.316222+0000","last_became_peered":"2024-04-24T01:55:30.316222+0000","last_unstale":"2024-04-24T01:56:46.378284+0000","last_undegraded":"2024-04-24T01:56:46.378284+0000","last_fullsized":"2024-04-24T01:56:46.378284+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":86,"log_dups_size":0,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":8,"snaptrim_duration":0,"stat_sum":{"num_bytes":11764900,"num_objects":6,"num_object_clones":2,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":85,"num_read_kb":50482,"num_write":74,"num_write_kb":20601,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":222,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-24T01:56:49.073565+0000","last_change":"2024-04-24T01:55:18.265905+0000","last_active":"2024-04-24T01:56:49.073565+0000","last_peered":"2024-04-24T01:56:49.073565+0000","last_clean":"2024-04-24T01:56:49.073565+0000","last_became_active":"2024-04-24T01:55:18.257020+0000","last_became_peered":"2024-04-24T01:55:18.257020+0000","last_unstale":"2024-04-24T01:56:49.073565+0000","last_undegraded":"2024-04-24T01:56:49.073565+0000","last_fullsized":"2024-04-24T01:56:49.073565+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"94'61","reported_seq":316,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-24T01:56:49.048145+0000","last_change":"2024-04-24T01:56:48.291437+0000","last_active":"2024-04-24T01:56:49.048145+0000","last_peered":"2024-04-24T01:56:49.048145+0000","last_clean":"2024-04-24T01:56:49.048145+0000","last_became_active":"2024-04-24T01:55:30.344129+0000","last_became_peered":"2024-04-24T01:55:30.344129+0000","last_unstale":"2024-04-24T01:56:49.048145+0000","last_undegraded":"2024-04-24T01:56:49.048145+0000","last_fullsized":"2024-04-24T01:56:49.048145+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0,"stat_sum":{"num_bytes":9462341,"num_objects":6,"num_object_clones":3,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":55,"num_read_kb":32627,"num_write":45,"num_write_kb":12400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":209,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-24T01:56:48.145239+0000","last_change":"2024-04-24T01:55:18.215251+0000","last_active":"2024-04-24T01:56:48.145239+0000","last_peered":"2024-04-24T01:56:48.145239+0000","last_clean":"2024-04-24T01:56:48.145239+0000","last_became_active":"2024-04-24T01:55:18.206605+0000","last_became_peered":"2024-04-24T01:55:18.206605+0000","last_unstale":"2024-04-24T01:56:48.145239+0000","last_undegraded":"2024-04-24T01:56:48.145239+0000","last_fullsized":"2024-04-24T01:56:48.145239+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"95'36","reported_seq":283,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-24T01:56:49.095061+0000","last_change":"2024-04-24T01:56:48.191162+0000","last_active":"2024-04-24T01:56:49.095061+0000","last_peered":"2024-04-24T01:56:49.095061+0000","last_clean":"2024-04-24T01:56:49.095061+0000","last_became_active":"2024-04-24T01:55:30.401366+0000","last_became_peered":"2024-04-24T01:55:30.401366+0000","last_unstale":"2024-04-24T01:56:49.095061+0000","last_undegraded":"2024-04-24T01:56:49.095061+0000","last_fullsized":"2024-04-24T01:56:49.095061+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0,"stat_sum":{"num_bytes":7176556,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":26,"num_read_kb":19277,"num_write":24,"num_write_kb":6694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":222,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-24T01:56:49.067242+0000","last_change":"2024-04-24T01:55:18.230893+0000","last_active":"2024-04-24T01:56:49.067242+0000","last_peered":"2024-04-24T01:56:49.067242+0000","last_clean":"2024-04-24T01:56:49.067242+0000","last_became_active":"2024-04-24T01:55:18.225188+0000","last_became_peered":"2024-04-24T01:55:18.225188+0000","last_unstale":"2024-04-24T01:56:49.067242+0000","last_undegraded":"2024-04-24T01:56:49.067242+0000","last_fullsized":"2024-04-24T01:56:49.067242+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"83'64","reported_seq":313,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-24T01:56:49.137017+0000","last_change":"2024-04-24T01:56:48.176683+0000","last_active":"2024-04-24T01:56:49.137017+0000","last_peered":"2024-04-24T01:56:49.137017+0000","last_clean":"2024-04-24T01:56:49.137017+0000","last_became_active":"2024-04-24T01:55:30.343318+0000","last_became_peered":"2024-04-24T01:55:30.343318+0000","last_unstale":"2024-04-24T01:56:49.137017+0000","last_undegraded":"2024-04-24T01:56:49.137017+0000","last_fullsized":"2024-04-24T01:56:49.137017+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":7,"snaptrim_duration":0,"stat_sum":{"num_bytes":6571504,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":47,"num_read_kb":36598,"num_write":46,"num_write_kb":13987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"92'87","reported_seq":328,"reported_epoch":94,"state":"active+clean+snaptrim","last_fresh":"2024-04-24T01:56:46.211937+0000","last_change":"2024-04-24T01:56:46.200543+0000","last_active":"2024-04-24T01:56:46.211937+0000","last_peered":"2024-04-24T01:56:46.211937+0000","last_clean":"2024-04-24T01:56:46.211937+0000","last_became_active":"2024-04-24T01:55:30.345308+0000","last_became_peered":"2024-04-24T01:55:30.345308+0000","last_unstale":"2024-04-24T01:56:46.211937+0000","last_undegraded":"2024-04-24T01:56:46.211937+0000","last_fullsized":"2024-04-24T01:56:46.211937+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":8,"snaptrim_duration":0,"stat_sum":{"num_bytes":7411887,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":72,"num_read_kb":40506,"num_write":59,"num_write_kb":15036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":214,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-24T01:56:49.152418+0000","last_change":"2024-04-24T01:55:18.224238+0000","last_active":"2024-04-24T01:56:49.152418+0000","last_peered":"2024-04-24T01:56:49.152418+0000","last_clean":"2024-04-24T01:56:49.152418+0000","last_became_active":"2024-04-24T01:55:18.219131+0000","last_became_peered":"2024-04-24T01:55:18.219131+0000","last_unstale":"2024-04-24T01:56:49.152418+0000","last_undegraded":"2024-04-24T01:56:49.152418+0000","last_fullsized":"2024-04-24T01:56:49.152418+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"92'43","reported_seq":284,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-24T01:56:49.143694+0000","last_change":"2024-04-24T01:56:48.183355+0000","last_active":"2024-04-24T01:56:49.143694+0000","last_peered":"2024-04-24T01:56:49.143694+0000","last_clean":"2024-04-24T01:56:49.143694+0000","last_became_active":"2024-04-24T01:55:30.319958+0000","last_became_peered":"2024-04-24T01:55:30.319958+0000","last_unstale":"2024-04-24T01:56:49.143694+0000","last_undegraded":"2024-04-24T01:56:49.143694+0000","last_fullsized":"2024-04-24T01:56:49.143694+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":22,"num_read_kb":19606,"num_write":28,"num_write_kb":7720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":216,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-24T01:56:49.061600+0000","last_change":"2024-04-24T01:55:18.235763+0000","last_active":"2024-04-24T01:56:49.061600+0000","last_peered":"2024-04-24T01:56:49.061600+0000","last_clean":"2024-04-24T01:56:49.061600+0000","last_became_active":"2024-04-24T01:55:18.229175+0000","last_became_peered":"2024-04-24T01:55:18.229175+0000","last_unstale":"2024-04-24T01:56:49.061600+0000","last_undegraded":"2024-04-24T01:56:49.061600+0000","last_fullsized":"2024-04-24T01:56:49.061600+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"68'28","reported_seq":279,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-24T01:56:49.044135+0000","last_change":"2024-04-24T01:56:48.189258+0000","last_active":"2024-04-24T01:56:49.044135+0000","last_peered":"2024-04-24T01:56:49.044135+0000","last_clean":"2024-04-24T01:56:49.044135+0000","last_became_active":"2024-04-24T01:55:30.372690+0000","last_became_peered":"2024-04-24T01:55:30.372690+0000","last_unstale":"2024-04-24T01:56:49.044135+0000","last_undegraded":"2024-04-24T01:56:49.044135+0000","last_fullsized":"2024-04-24T01:56:49.044135+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":3,"snaptrim_duration":0,"stat_sum":{"num_bytes":2521050,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":28,"num_read_kb":16701,"num_write":24,"num_write_kb":6609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":214,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-24T01:56:49.141807+0000","last_change":"2024-04-24T01:55:18.278351+0000","last_active":"2024-04-24T01:56:49.141807+0000","last_peered":"2024-04-24T01:56:49.141807+0000","last_clean":"2024-04-24T01:56:49.141807+0000","last_became_active":"2024-04-24T01:55:18.273668+0000","last_became_peered":"2024-04-24T01:55:18.273668+0000","last_unstale":"2024-04-24T01:56:49.141807+0000","last_undegraded":"2024-04-24T01:56:49.141807+0000","last_fullsized":"2024-04-24T01:56:49.141807+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"94'81","reported_seq":328,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-24T01:56:49.151440+0000","last_change":"2024-04-24T01:56:48.220004+0000","last_active":"2024-04-24T01:56:49.151440+0000","last_peered":"2024-04-24T01:56:49.151440+0000","last_clean":"2024-04-24T01:56:49.151440+0000","last_became_active":"2024-04-24T01:55:30.363123+0000","last_became_peered":"2024-04-24T01:55:30.363123+0000","last_unstale":"2024-04-24T01:56:49.151440+0000","last_undegraded":"2024-04-24T01:56:49.151440+0000","last_fullsized":"2024-04-24T01:56:49.151440+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":7,"snaptrim_duration":0,"stat_sum":{"num_bytes":5865643,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":49,"num_read_kb":25820,"num_write":63,"num_write_kb":15734,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":206,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-24T01:56:46.199955+0000","last_change":"2024-04-24T01:55:18.241117+0000","last_active":"2024-04-24T01:56:46.199955+0000","last_peered":"2024-04-24T01:56:46.199955+0000","last_clean":"2024-04-24T01:56:46.199955+0000","last_became_active":"2024-04-24T01:55:18.234313+0000","last_became_peered":"2024-04-24T01:55:18.234313+0000","last_unstale":"2024-04-24T01:56:46.199955+0000","last_undegraded":"2024-04-24T01:56:46.199955+0000","last_fullsized":"2024-04-24T01:56:46.199955+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":101841832,"num_objects":53,"num_object_clones":14,"num_object_copies":159,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":734,"num_read_kb":450558,"num_write":704,"num_write_kb":191952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":218742784,"data_stored":230125098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":940,"ondisk_log_size":940,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":1}],"osd_stats":[{"osd":3,"up_from":10,"seq":42949673054,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":90984,"kb_used_data":59044,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94280856,"statfs":{"total":96636764160,"available":96543596544,"internally_reserved":0,"allocated":60461056,"data_stored":64042371,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673054,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":70744,"kb_used_data":43920,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94301096,"statfs":{"total":96636764160,"available":96564322304,"internally_reserved":0,"allocated":44974080,"data_stored":46189718,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705759,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":73784,"kb_used_data":46960,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94298056,"statfs":{"total":96636764160,"available":96561209344,"internally_reserved":0,"allocated":48087040,"data_stored":50297287,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705758,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":83808,"kb_used_data":56984,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94288032,"statfs":{"total":96636764160,"available":96550944768,"internally_reserved":0,"allocated":58351616,"data_stored":61735965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":50176000,"data_stored":52569413,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57593856,"data_stored":61422925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":48099328,"data_stored":49493619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":62873600,"data_stored":66639141,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T01:56:52.424 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T01:56:52.532 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:52.527+0000 7f00f099d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:52.535 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:52.534+0000 7f00f099d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:52.719 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:56:52.720 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-24T01:56:52.732 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":138,"stamp":"2024-04-24T01:56:52.041056+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":102432200,"num_objects":55,"num_object_clones":14,"num_object_copies":165,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":837,"num_read_kb":450649,"num_write":841,"num_write_kb":193846,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1037,"ondisk_log_size":1037,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":319320,"kb_used_data":206908,"kb_used_omap":0,"kb_used_meta":111424,"kb_avail":377168040,"statfs":{"total":386547056640,"available":386220072960,"internally_reserved":0,"allocated":211873792,"data_stored":222265341,"data_compressed":89889,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":114098176},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":8798445,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":-1,"num_read":25,"num_read_kb":18119,"num_write":28,"num_write_kb":9057,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.011846"},"pg_stats":[{"pgid":"3.f","version":"75'32","reported_seq":292,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-24T01:56:49.057838+0000","last_change":"2024-04-24T01:56:48.058716+0000","last_active":"2024-04-24T01:56:49.057838+0000","last_peered":"2024-04-24T01:56:49.057838+0000","last_clean":"2024-04-24T01:56:49.057838+0000","last_became_active":"2024-04-24T01:55:30.386726+0000","last_became_peered":"2024-04-24T01:55:30.386726+0000","last_unstale":"2024-04-24T01:56:49.057838+0000","last_undegraded":"2024-04-24T01:56:49.057838+0000","last_fullsized":"2024-04-24T01:56:49.057838+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0,"stat_sum":{"num_bytes":5361433,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":39,"num_read_kb":28312,"num_write":22,"num_write_kb":6376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"92'95","reported_seq":346,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-24T01:56:49.144135+0000","last_change":"2024-04-24T01:56:48.175633+0000","last_active":"2024-04-24T01:56:49.144135+0000","last_peered":"2024-04-24T01:56:49.144135+0000","last_clean":"2024-04-24T01:56:49.144135+0000","last_became_active":"2024-04-24T01:55:30.321866+0000","last_became_peered":"2024-04-24T01:55:30.321866+0000","last_unstale":"2024-04-24T01:56:49.144135+0000","last_undegraded":"2024-04-24T01:56:49.144135+0000","last_fullsized":"2024-04-24T01:56:49.144135+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":7,"snaptrim_duration":0,"stat_sum":{"num_bytes":10437114,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":65,"num_read_kb":37555,"num_write":66,"num_write_kb":17834,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"92'75","reported_seq":307,"reported_epoch":94,"state":"active+clean+snaptrim","last_fresh":"2024-04-24T01:56:48.150894+0000","last_change":"2024-04-24T01:56:46.170323+0000","last_active":"2024-04-24T01:56:48.150894+0000","last_peered":"2024-04-24T01:56:48.150894+0000","last_clean":"2024-04-24T01:56:48.150894+0000","last_became_active":"2024-04-24T01:55:30.320186+0000","last_became_peered":"2024-04-24T01:55:30.320186+0000","last_unstale":"2024-04-24T01:56:48.150894+0000","last_undegraded":"2024-04-24T01:56:48.150894+0000","last_fullsized":"2024-04-24T01:56:48.150894+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":8,"snaptrim_duration":0,"stat_sum":{"num_bytes":5411854,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":12901,"num_write":57,"num_write_kb":14728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"95'72","reported_seq":332,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-24T01:56:49.116649+0000","last_change":"2024-04-24T01:56:48.058499+0000","last_active":"2024-04-24T01:56:49.116649+0000","last_peered":"2024-04-24T01:56:49.116649+0000","last_clean":"2024-04-24T01:56:49.116649+0000","last_became_active":"2024-04-24T01:55:30.288081+0000","last_became_peered":"2024-04-24T01:55:30.288081+0000","last_unstale":"2024-04-24T01:56:49.116649+0000","last_undegraded":"2024-04-24T01:56:49.116649+0000","last_fullsized":"2024-04-24T01:56:49.116649+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":9,"snaptrim_duration":0,"stat_sum":{"num_bytes":8134588,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":66,"num_read_kb":49021,"num_write":46,"num_write_kb":13369,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"94'35","reported_seq":286,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-24T01:56:49.062621+0000","last_change":"2024-04-24T01:56:48.251963+0000","last_active":"2024-04-24T01:56:49.062621+0000","last_peered":"2024-04-24T01:56:49.062621+0000","last_clean":"2024-04-24T01:56:49.062621+0000","last_became_active":"2024-04-24T01:55:30.333815+0000","last_became_peered":"2024-04-24T01:55:30.333815+0000","last_unstale":"2024-04-24T01:56:49.062621+0000","last_undegraded":"2024-04-24T01:56:49.062621+0000","last_fullsized":"2024-04-24T01:56:49.062621+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0,"stat_sum":{"num_bytes":3226944,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":30,"num_read_kb":17555,"num_write":31,"num_write_kb":8358,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"91'119","reported_seq":392,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-24T01:56:49.062340+0000","last_change":"2024-04-24T01:56:48.058276+0000","last_active":"2024-04-24T01:56:49.062340+0000","last_peered":"2024-04-24T01:56:49.062340+0000","last_clean":"2024-04-24T01:56:49.062340+0000","last_became_active":"2024-04-24T01:55:30.315075+0000","last_became_peered":"2024-04-24T01:55:30.315075+0000","last_unstale":"2024-04-24T01:56:49.062340+0000","last_undegraded":"2024-04-24T01:56:49.062340+0000","last_fullsized":"2024-04-24T01:56:49.062340+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":15,"snaptrim_duration":0,"stat_sum":{"num_bytes":16117913,"num_objects":7,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":100,"num_read_kb":58593,"num_write":97,"num_write_kb":26944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"95'20","reported_seq":271,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-24T01:56:49.101772+0000","last_change":"2024-04-24T01:56:48.063986+0000","last_active":"2024-04-24T01:56:49.101772+0000","last_peered":"2024-04-24T01:56:49.101772+0000","last_clean":"2024-04-24T01:56:49.101772+0000","last_became_active":"2024-04-24T01:55:30.350300+0000","last_became_peered":"2024-04-24T01:55:30.350300+0000","last_unstale":"2024-04-24T01:56:49.101772+0000","last_undegraded":"2024-04-24T01:56:49.101772+0000","last_fullsized":"2024-04-24T01:56:49.101772+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0,"stat_sum":{"num_bytes":2378105,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":13,"num_read_kb":5003,"num_write":17,"num_write_kb":4714,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"64'6","reported_seq":237,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-24T01:56:46.272687+0000","last_change":"2024-04-24T01:56:46.272687+0000","last_active":"2024-04-24T01:56:46.272687+0000","last_peered":"2024-04-24T01:56:46.272687+0000","last_clean":"2024-04-24T01:56:46.272687+0000","last_became_active":"2024-04-24T01:55:30.376813+0000","last_became_peered":"2024-04-24T01:55:30.376813+0000","last_unstale":"2024-04-24T01:56:46.272687+0000","last_undegraded":"2024-04-24T01:56:46.272687+0000","last_fullsized":"2024-04-24T01:56:46.272687+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'97","reported_seq":349,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-24T01:56:49.047435+0000","last_change":"2024-04-24T01:55:16.196470+0000","last_active":"2024-04-24T01:56:49.047435+0000","last_peered":"2024-04-24T01:56:49.047435+0000","last_clean":"2024-04-24T01:56:49.047435+0000","last_became_active":"2024-04-24T01:55:16.190464+0000","last_became_peered":"2024-04-24T01:55:16.190464+0000","last_unstale":"2024-04-24T01:56:49.047435+0000","last_undegraded":"2024-04-24T01:56:49.047435+0000","last_fullsized":"2024-04-24T01:56:49.047435+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:15.127115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:15.127115+0000","last_clean_scrub_stamp":"2024-04-24T01:55:15.127115+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":206,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-24T01:56:46.188874+0000","last_change":"2024-04-24T01:55:18.236148+0000","last_active":"2024-04-24T01:56:46.188874+0000","last_peered":"2024-04-24T01:56:46.188874+0000","last_clean":"2024-04-24T01:56:46.188874+0000","last_became_active":"2024-04-24T01:55:18.230154+0000","last_became_peered":"2024-04-24T01:55:18.230154+0000","last_unstale":"2024-04-24T01:56:46.188874+0000","last_undegraded":"2024-04-24T01:56:46.188874+0000","last_fullsized":"2024-04-24T01:56:46.188874+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"92'86","reported_seq":344,"reported_epoch":94,"state":"active+clean+snaptrim","last_fresh":"2024-04-24T01:56:46.378284+0000","last_change":"2024-04-24T01:56:46.197594+0000","last_active":"2024-04-24T01:56:46.378284+0000","last_peered":"2024-04-24T01:56:46.378284+0000","last_clean":"2024-04-24T01:56:46.378284+0000","last_became_active":"2024-04-24T01:55:30.316222+0000","last_became_peered":"2024-04-24T01:55:30.316222+0000","last_unstale":"2024-04-24T01:56:46.378284+0000","last_undegraded":"2024-04-24T01:56:46.378284+0000","last_fullsized":"2024-04-24T01:56:46.378284+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":86,"log_dups_size":0,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":8,"snaptrim_duration":0,"stat_sum":{"num_bytes":11764900,"num_objects":6,"num_object_clones":2,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":85,"num_read_kb":50482,"num_write":74,"num_write_kb":20601,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":222,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-24T01:56:49.073565+0000","last_change":"2024-04-24T01:55:18.265905+0000","last_active":"2024-04-24T01:56:49.073565+0000","last_peered":"2024-04-24T01:56:49.073565+0000","last_clean":"2024-04-24T01:56:49.073565+0000","last_became_active":"2024-04-24T01:55:18.257020+0000","last_became_peered":"2024-04-24T01:55:18.257020+0000","last_unstale":"2024-04-24T01:56:49.073565+0000","last_undegraded":"2024-04-24T01:56:49.073565+0000","last_fullsized":"2024-04-24T01:56:49.073565+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"94'61","reported_seq":316,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-24T01:56:49.048145+0000","last_change":"2024-04-24T01:56:48.291437+0000","last_active":"2024-04-24T01:56:49.048145+0000","last_peered":"2024-04-24T01:56:49.048145+0000","last_clean":"2024-04-24T01:56:49.048145+0000","last_became_active":"2024-04-24T01:55:30.344129+0000","last_became_peered":"2024-04-24T01:55:30.344129+0000","last_unstale":"2024-04-24T01:56:49.048145+0000","last_undegraded":"2024-04-24T01:56:49.048145+0000","last_fullsized":"2024-04-24T01:56:49.048145+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0,"stat_sum":{"num_bytes":9462341,"num_objects":6,"num_object_clones":3,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":55,"num_read_kb":32627,"num_write":45,"num_write_kb":12400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":209,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-24T01:56:48.145239+0000","last_change":"2024-04-24T01:55:18.215251+0000","last_active":"2024-04-24T01:56:48.145239+0000","last_peered":"2024-04-24T01:56:48.145239+0000","last_clean":"2024-04-24T01:56:48.145239+0000","last_became_active":"2024-04-24T01:55:18.206605+0000","last_became_peered":"2024-04-24T01:55:18.206605+0000","last_unstale":"2024-04-24T01:56:48.145239+0000","last_undegraded":"2024-04-24T01:56:48.145239+0000","last_fullsized":"2024-04-24T01:56:48.145239+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"95'36","reported_seq":283,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-24T01:56:49.095061+0000","last_change":"2024-04-24T01:56:48.191162+0000","last_active":"2024-04-24T01:56:49.095061+0000","last_peered":"2024-04-24T01:56:49.095061+0000","last_clean":"2024-04-24T01:56:49.095061+0000","last_became_active":"2024-04-24T01:55:30.401366+0000","last_became_peered":"2024-04-24T01:55:30.401366+0000","last_unstale":"2024-04-24T01:56:49.095061+0000","last_undegraded":"2024-04-24T01:56:49.095061+0000","last_fullsized":"2024-04-24T01:56:49.095061+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0,"stat_sum":{"num_bytes":7176556,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":26,"num_read_kb":19277,"num_write":24,"num_write_kb":6694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":222,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-24T01:56:49.067242+0000","last_change":"2024-04-24T01:55:18.230893+0000","last_active":"2024-04-24T01:56:49.067242+0000","last_peered":"2024-04-24T01:56:49.067242+0000","last_clean":"2024-04-24T01:56:49.067242+0000","last_became_active":"2024-04-24T01:55:18.225188+0000","last_became_peered":"2024-04-24T01:55:18.225188+0000","last_unstale":"2024-04-24T01:56:49.067242+0000","last_undegraded":"2024-04-24T01:56:49.067242+0000","last_fullsized":"2024-04-24T01:56:49.067242+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"83'64","reported_seq":313,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-24T01:56:49.137017+0000","last_change":"2024-04-24T01:56:48.176683+0000","last_active":"2024-04-24T01:56:49.137017+0000","last_peered":"2024-04-24T01:56:49.137017+0000","last_clean":"2024-04-24T01:56:49.137017+0000","last_became_active":"2024-04-24T01:55:30.343318+0000","last_became_peered":"2024-04-24T01:55:30.343318+0000","last_unstale":"2024-04-24T01:56:49.137017+0000","last_undegraded":"2024-04-24T01:56:49.137017+0000","last_fullsized":"2024-04-24T01:56:49.137017+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":7,"snaptrim_duration":0,"stat_sum":{"num_bytes":6571504,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":47,"num_read_kb":36598,"num_write":46,"num_write_kb":13987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"92'87","reported_seq":328,"reported_epoch":94,"state":"active+clean+snaptrim","last_fresh":"2024-04-24T01:56:46.211937+0000","last_change":"2024-04-24T01:56:46.200543+0000","last_active":"2024-04-24T01:56:46.211937+0000","last_peered":"2024-04-24T01:56:46.211937+0000","last_clean":"2024-04-24T01:56:46.211937+0000","last_became_active":"2024-04-24T01:55:30.345308+0000","last_became_peered":"2024-04-24T01:55:30.345308+0000","last_unstale":"2024-04-24T01:56:46.211937+0000","last_undegraded":"2024-04-24T01:56:46.211937+0000","last_fullsized":"2024-04-24T01:56:46.211937+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":8,"snaptrim_duration":0,"stat_sum":{"num_bytes":7411887,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":72,"num_read_kb":40506,"num_write":59,"num_write_kb":15036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":214,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-24T01:56:49.152418+0000","last_change":"2024-04-24T01:55:18.224238+0000","last_active":"2024-04-24T01:56:49.152418+0000","last_peered":"2024-04-24T01:56:49.152418+0000","last_clean":"2024-04-24T01:56:49.152418+0000","last_became_active":"2024-04-24T01:55:18.219131+0000","last_became_peered":"2024-04-24T01:55:18.219131+0000","last_unstale":"2024-04-24T01:56:49.152418+0000","last_undegraded":"2024-04-24T01:56:49.152418+0000","last_fullsized":"2024-04-24T01:56:49.152418+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"92'43","reported_seq":284,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-24T01:56:49.143694+0000","last_change":"2024-04-24T01:56:48.183355+0000","last_active":"2024-04-24T01:56:49.143694+0000","last_peered":"2024-04-24T01:56:49.143694+0000","last_clean":"2024-04-24T01:56:49.143694+0000","last_became_active":"2024-04-24T01:55:30.319958+0000","last_became_peered":"2024-04-24T01:55:30.319958+0000","last_unstale":"2024-04-24T01:56:49.143694+0000","last_undegraded":"2024-04-24T01:56:49.143694+0000","last_fullsized":"2024-04-24T01:56:49.143694+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":22,"num_read_kb":19606,"num_write":28,"num_write_kb":7720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":216,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-24T01:56:49.061600+0000","last_change":"2024-04-24T01:55:18.235763+0000","last_active":"2024-04-24T01:56:49.061600+0000","last_peered":"2024-04-24T01:56:49.061600+0000","last_clean":"2024-04-24T01:56:49.061600+0000","last_became_active":"2024-04-24T01:55:18.229175+0000","last_became_peered":"2024-04-24T01:55:18.229175+0000","last_unstale":"2024-04-24T01:56:49.061600+0000","last_undegraded":"2024-04-24T01:56:49.061600+0000","last_fullsized":"2024-04-24T01:56:49.061600+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"68'28","reported_seq":279,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-24T01:56:49.044135+0000","last_change":"2024-04-24T01:56:48.189258+0000","last_active":"2024-04-24T01:56:49.044135+0000","last_peered":"2024-04-24T01:56:49.044135+0000","last_clean":"2024-04-24T01:56:49.044135+0000","last_became_active":"2024-04-24T01:55:30.372690+0000","last_became_peered":"2024-04-24T01:55:30.372690+0000","last_unstale":"2024-04-24T01:56:49.044135+0000","last_undegraded":"2024-04-24T01:56:49.044135+0000","last_fullsized":"2024-04-24T01:56:49.044135+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":3,"snaptrim_duration":0,"stat_sum":{"num_bytes":2521050,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":28,"num_read_kb":16701,"num_write":24,"num_write_kb":6609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":214,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-24T01:56:49.141807+0000","last_change":"2024-04-24T01:55:18.278351+0000","last_active":"2024-04-24T01:56:49.141807+0000","last_peered":"2024-04-24T01:56:49.141807+0000","last_clean":"2024-04-24T01:56:49.141807+0000","last_became_active":"2024-04-24T01:55:18.273668+0000","last_became_peered":"2024-04-24T01:55:18.273668+0000","last_unstale":"2024-04-24T01:56:49.141807+0000","last_undegraded":"2024-04-24T01:56:49.141807+0000","last_fullsized":"2024-04-24T01:56:49.141807+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"94'81","reported_seq":328,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-24T01:56:49.151440+0000","last_change":"2024-04-24T01:56:48.220004+0000","last_active":"2024-04-24T01:56:49.151440+0000","last_peered":"2024-04-24T01:56:49.151440+0000","last_clean":"2024-04-24T01:56:49.151440+0000","last_became_active":"2024-04-24T01:55:30.363123+0000","last_became_peered":"2024-04-24T01:55:30.363123+0000","last_unstale":"2024-04-24T01:56:49.151440+0000","last_undegraded":"2024-04-24T01:56:49.151440+0000","last_fullsized":"2024-04-24T01:56:49.151440+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":7,"snaptrim_duration":0,"stat_sum":{"num_bytes":5865643,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":49,"num_read_kb":25820,"num_write":63,"num_write_kb":15734,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":206,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-24T01:56:46.199955+0000","last_change":"2024-04-24T01:55:18.241117+0000","last_active":"2024-04-24T01:56:46.199955+0000","last_peered":"2024-04-24T01:56:46.199955+0000","last_clean":"2024-04-24T01:56:46.199955+0000","last_became_active":"2024-04-24T01:55:18.234313+0000","last_became_peered":"2024-04-24T01:55:18.234313+0000","last_unstale":"2024-04-24T01:56:46.199955+0000","last_undegraded":"2024-04-24T01:56:46.199955+0000","last_fullsized":"2024-04-24T01:56:46.199955+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":101841832,"num_objects":53,"num_object_clones":14,"num_object_copies":159,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":734,"num_read_kb":450558,"num_write":704,"num_write_kb":191952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":218742784,"data_stored":230125098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":940,"ondisk_log_size":940,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":1}],"osd_stats":[{"osd":3,"up_from":10,"seq":42949673054,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":90984,"kb_used_data":59044,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94280856,"statfs":{"total":96636764160,"available":96543596544,"internally_reserved":0,"allocated":60461056,"data_stored":64042371,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673054,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":70744,"kb_used_data":43920,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94301096,"statfs":{"total":96636764160,"available":96564322304,"internally_reserved":0,"allocated":44974080,"data_stored":46189718,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705759,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":73784,"kb_used_data":46960,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94298056,"statfs":{"total":96636764160,"available":96561209344,"internally_reserved":0,"allocated":48087040,"data_stored":50297287,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705758,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":83808,"kb_used_data":56984,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94288032,"statfs":{"total":96636764160,"available":96550944768,"internally_reserved":0,"allocated":58351616,"data_stored":61735965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":50176000,"data_stored":52569413,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57593856,"data_stored":61422925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":48099328,"data_stored":49493619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":62873600,"data_stored":66639141,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T01:56:52.734 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-04-24T01:56:52.734 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-04-24T01:56:52.734 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-04-24T01:56:52.734 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-04-24T01:56:52.841 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:52.836+0000 7ff7d095d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:52.845 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:52.845+0000 7ff7d095d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:52.860 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T01:56:52.860 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T01:56:52.860 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T01:56:52.860 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T01:56:52.860 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T01:56:52.860 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T01:56:52.860 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T01:56:52.860 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T01:56:52.860 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T01:56:52.860 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T01:56:52.860 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T01:56:52.860 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T01:56:52.863 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T01:56:52.863 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T01:56:52.864 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-04-24T01:56:52.924 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T01:56:52.973 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:52.967+0000 7f890262d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:52.977 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:52.976+0000 7f890262d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:52.992 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T01:56:52.992 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T01:56:52.992 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T01:56:52.992 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T01:56:52.992 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T01:56:52.992 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T01:56:52.992 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T01:56:52.992 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T01:56:52.992 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T01:56:52.992 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T01:56:52.992 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T01:56:52.992 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T01:56:52.996 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T01:56:52.996 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T01:56:52.996 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-04-24T01:56:53.033 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:53.031+0000 7f54048ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:53.037 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:53.036+0000 7f54048ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:53.044 INFO:tasks.rados.rados.0.smithi079.stdout:382: done (0 left) 2024-04-24T01:56:53.044 INFO:tasks.rados.rados.0.smithi079.stdout:383: write (excl) oid 17 current snap is 43 2024-04-24T01:56:53.044 INFO:tasks.rados.rados.0.smithi079.stdout:383: seq_num 202 ranges {595014=625476,1820892=523392,2797915=596476} 2024-04-24T01:56:53.053 INFO:tasks.rados.rados.0.smithi079.stdout:383: writing smithi07929762-17 from 595014 to 1220490 tid 1 2024-04-24T01:56:53.062 INFO:tasks.rados.rados.0.smithi079.stdout:383: writing smithi07929762-17 from 1820892 to 2344284 tid 2 2024-04-24T01:56:53.071 INFO:tasks.rados.rados.0.smithi079.stdout:383: writing smithi07929762-17 from 2797915 to 3394391 tid 3 2024-04-24T01:56:53.071 INFO:tasks.rados.rados.0.smithi079.stdout:384: read oid 20 snap -1 2024-04-24T01:56:53.072 INFO:tasks.rados.rados.0.smithi079.stdout:384: expect (ObjNum 154 snap 21 seq_num 154) 2024-04-24T01:56:53.072 INFO:tasks.rados.rados.0.smithi079.stdout:385: delete oid 42 current snap is 43 2024-04-24T01:56:53.142 INFO:tasks.rados.rados.0.smithi079.stdout:383: finishing write tid 1 to smithi07929762-17 2024-04-24T01:56:53.144 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:53.134+0000 7fc6ec4ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:53.151 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:53.150+0000 7fc6ec4ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:53.155 INFO:tasks.rados.rados.0.smithi079.stdout:385: done (2 left) 2024-04-24T01:56:53.155 INFO:tasks.rados.rados.0.smithi079.stdout:386: read oid 12 snap 41 2024-04-24T01:56:53.155 INFO:tasks.rados.rados.0.smithi079.stdout:386: expect (ObjNum 198 snap 40 seq_num 198) 2024-04-24T01:56:53.155 INFO:tasks.rados.rados.0.smithi079.stdout:387: read oid 46 snap -1 2024-04-24T01:56:53.155 INFO:tasks.rados.rados.0.smithi079.stdout:387: expect deleted 2024-04-24T01:56:53.155 INFO:tasks.rados.rados.0.smithi079.stdout:388: read oid 47 snap -1 2024-04-24T01:56:53.155 INFO:tasks.rados.rados.0.smithi079.stdout:388: expect (ObjNum 193 snap 38 seq_num 193) 2024-04-24T01:56:53.155 INFO:tasks.rados.rados.0.smithi079.stdout:389: read oid 26 snap -1 2024-04-24T01:56:53.155 INFO:tasks.rados.rados.0.smithi079.stdout:389: expect (ObjNum 181 snap 33 seq_num 181) 2024-04-24T01:56:53.155 INFO:tasks.rados.rados.0.smithi079.stdout:390: read oid 43 snap -1 2024-04-24T01:56:53.156 INFO:tasks.rados.rados.0.smithi079.stdout:390: expect deleted 2024-04-24T01:56:53.156 INFO:tasks.rados.rados.0.smithi079.stdout:391: snap_create 2024-04-24T01:56:53.156 INFO:tasks.rados.rados.0.smithi079.stdout:384: expect (ObjNum 154 snap 21 seq_num 154) 2024-04-24T01:56:53.182 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T01:56:53.183 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T01:56:53.183 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T01:56:53.183 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T01:56:53.183 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T01:56:53.183 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T01:56:53.183 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T01:56:53.183 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T01:56:53.183 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T01:56:53.183 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T01:56:53.183 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T01:56:53.183 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T01:56:53.186 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T01:56:53.187 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T01:56:53.187 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-04-24T01:56:53.319 INFO:tasks.rados.rados.0.smithi079.stdout:388: expect (ObjNum 193 snap 38 seq_num 193) 2024-04-24T01:56:53.337 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:53.331+0000 7f0f7426d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:53.340 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:53.340+0000 7f0f7426d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:53.361 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T01:56:53.361 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T01:56:53.361 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T01:56:53.361 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T01:56:53.361 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T01:56:53.361 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T01:56:53.361 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T01:56:53.361 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T01:56:53.361 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T01:56:53.361 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T01:56:53.361 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T01:56:53.361 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T01:56:53.365 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T01:56:53.365 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T01:56:53.505 INFO:tasks.rados.rados.0.smithi079.stdout:383: finishing write tid 2 to smithi07929762-17 2024-04-24T01:56:53.506 INFO:tasks.rados.rados.0.smithi079.stdout:383: finishing write tid 3 to smithi07929762-17 2024-04-24T01:56:53.506 INFO:tasks.rados.rados.0.smithi079.stdout:389: expect (ObjNum 181 snap 33 seq_num 181) 2024-04-24T01:56:53.620 INFO:tasks.rados.rados.0.smithi079.stdout:383: finishing write tid 4 to smithi07929762-17 2024-04-24T01:56:53.620 INFO:tasks.rados.rados.0.smithi079.stdout:383: finishing write tid 5 to smithi07929762-17 2024-04-24T01:56:53.620 INFO:tasks.rados.rados.0.smithi079.stdout:383: oid 17 updating version 0 to 94 2024-04-24T01:56:53.620 INFO:tasks.rados.rados.0.smithi079.stdout:383: oid 17 version 94 is already newer than 91 2024-04-24T01:56:53.620 INFO:tasks.rados.rados.0.smithi079.stdout:383: oid 17 version 94 is already newer than 94 2024-04-24T01:56:53.620 INFO:tasks.rados.rados.0.smithi079.stdout:383: oid 17 version 94 is already newer than 94 2024-04-24T01:56:53.620 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 17 v 94 (ObjNum 202 snap 43 seq_num 202) dirty exists 2024-04-24T01:56:53.620 INFO:tasks.rados.rados.0.smithi079.stdout:383: left oid 17 (ObjNum 202 snap 43 seq_num 202) 2024-04-24T01:56:53.620 INFO:tasks.rados.rados.0.smithi079.stdout:386: expect (ObjNum 198 snap 40 seq_num 198) 2024-04-24T01:56:54.054 INFO:tasks.rados.rados.0.smithi079.stdout:383: done (7 left) 2024-04-24T01:56:54.055 INFO:tasks.rados.rados.0.smithi079.stdout:384: done (6 left) 2024-04-24T01:56:54.055 INFO:tasks.rados.rados.0.smithi079.stdout:386: done (5 left) 2024-04-24T01:56:54.055 INFO:tasks.rados.rados.0.smithi079.stdout:387: done (4 left) 2024-04-24T01:56:54.055 INFO:tasks.rados.rados.0.smithi079.stdout:388: done (3 left) 2024-04-24T01:56:54.055 INFO:tasks.rados.rados.0.smithi079.stdout:389: done (2 left) 2024-04-24T01:56:54.055 INFO:tasks.rados.rados.0.smithi079.stdout:390: done (1 left) 2024-04-24T01:56:54.055 INFO:tasks.rados.rados.0.smithi079.stdout:391: done (0 left) 2024-04-24T01:56:54.055 INFO:tasks.rados.rados.0.smithi079.stdout:392: read oid 26 snap 37 2024-04-24T01:56:54.055 INFO:tasks.rados.rados.0.smithi079.stdout:392: expect (ObjNum 181 snap 33 seq_num 181) 2024-04-24T01:56:54.055 INFO:tasks.rados.rados.0.smithi079.stdout:393: delete oid 29 current snap is 44 2024-04-24T01:56:54.115 INFO:tasks.rados.rados.0.smithi079.stdout:392: expect (ObjNum 181 snap 33 seq_num 181) 2024-04-24T01:56:54.236 INFO:tasks.rados.rados.0.smithi079.stdout:392: done (1 left) 2024-04-24T01:56:54.236 INFO:tasks.rados.rados.0.smithi079.stdout:393: done (0 left) 2024-04-24T01:56:54.236 INFO:tasks.rados.rados.0.smithi079.stdout:394: write oid 4 current snap is 44 2024-04-24T01:56:54.236 INFO:tasks.rados.rados.0.smithi079.stdout:394: seq_num 205 ranges {645435=451223,1879985=787072,3280184=165251} 2024-04-24T01:56:54.242 INFO:tasks.rados.rados.0.smithi079.stdout:394: writing smithi07929762-4 from 645435 to 1096658 tid 1 2024-04-24T01:56:54.253 INFO:tasks.rados.rados.0.smithi079.stdout:394: writing smithi07929762-4 from 1879985 to 2667057 tid 2 2024-04-24T01:56:54.265 INFO:tasks.rados.rados.0.smithi079.stdout:394: writing smithi07929762-4 from 3280184 to 3445435 tid 3 2024-04-24T01:56:54.265 INFO:tasks.rados.rados.0.smithi079.stdout:395: read oid 29 snap -1 2024-04-24T01:56:54.265 INFO:tasks.rados.rados.0.smithi079.stdout:395: expect deleted 2024-04-24T01:56:54.265 INFO:tasks.rados.rados.0.smithi079.stdout:396: read oid 44 snap -1 2024-04-24T01:56:54.265 INFO:tasks.rados.rados.0.smithi079.stdout:396: expect (ObjNum 105 snap 9 seq_num 105) 2024-04-24T01:56:54.265 INFO:tasks.rados.rados.0.smithi079.stdout:397: write (excl) oid 23 current snap is 44 2024-04-24T01:56:54.265 INFO:tasks.rados.rados.0.smithi079.stdout:397: seq_num 206 ranges {662242=642825,2054820=530730,3307607=154635} 2024-04-24T01:56:54.280 INFO:tasks.rados.rados.0.smithi079.stdout:397: writing smithi07929762-23 from 662242 to 1305067 tid 1 2024-04-24T01:56:54.290 INFO:tasks.rados.rados.0.smithi079.stdout:397: writing smithi07929762-23 from 2054820 to 2585550 tid 2 2024-04-24T01:56:54.295 INFO:tasks.rados.rados.0.smithi079.stdout:397: writing smithi07929762-23 from 3307607 to 3462242 tid 3 2024-04-24T01:56:54.295 INFO:tasks.rados.rados.0.smithi079.stdout:398: delete oid 31 current snap is 44 2024-04-24T01:56:54.297 INFO:tasks.rados.rados.0.smithi079.stdout:394: finishing write tid 1 to smithi07929762-4 2024-04-24T01:56:54.302 INFO:tasks.rados.rados.0.smithi079.stdout:396: expect (ObjNum 105 snap 9 seq_num 105) 2024-04-24T01:56:54.544 INFO:tasks.rados.rados.0.smithi079.stdout:394: finishing write tid 2 to smithi07929762-4 2024-04-24T01:56:54.544 INFO:tasks.rados.rados.0.smithi079.stdout:394: finishing write tid 3 to smithi07929762-4 2024-04-24T01:56:54.544 INFO:tasks.rados.rados.0.smithi079.stdout:394: finishing write tid 4 to smithi07929762-4 2024-04-24T01:56:54.544 INFO:tasks.rados.rados.0.smithi079.stdout:395: done (4 left) 2024-04-24T01:56:54.544 INFO:tasks.rados.rados.0.smithi079.stdout:396: done (3 left) 2024-04-24T01:56:54.544 INFO:tasks.rados.rados.0.smithi079.stdout:398: done (2 left) 2024-04-24T01:56:54.544 INFO:tasks.rados.rados.0.smithi079.stdout:399: write oid 48 current snap is 44 2024-04-24T01:56:54.544 INFO:tasks.rados.rados.0.smithi079.stdout:394: finishing write tid 5 to smithi07929762-4 2024-04-24T01:56:54.544 INFO:tasks.rados.rados.0.smithi079.stdout:394: oid 4 updating version 0 to 28 2024-04-24T01:56:54.544 INFO:tasks.rados.rados.0.smithi079.stdout:394: oid 4 version 28 is already newer than 28 2024-04-24T01:56:54.545 INFO:tasks.rados.rados.0.smithi079.stdout:394: oid 4 version 28 is already newer than 28 2024-04-24T01:56:54.545 INFO:tasks.rados.rados.0.smithi079.stdout:394: oid 4 version 28 is already newer than 28 2024-04-24T01:56:54.545 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 4 v 28 (ObjNum 205 snap 44 seq_num 205) dirty exists 2024-04-24T01:56:54.545 INFO:tasks.rados.rados.0.smithi079.stdout:394: left oid 4 (ObjNum 205 snap 44 seq_num 205) 2024-04-24T01:56:54.545 INFO:tasks.rados.rados.0.smithi079.stdout:397: finishing write tid 1 to smithi07929762-23 2024-04-24T01:56:54.545 INFO:tasks.rados.rados.0.smithi079.stdout:397: finishing write tid 2 to smithi07929762-23 2024-04-24T01:56:54.545 INFO:tasks.rados.rados.0.smithi079.stdout:399: seq_num 208 ranges {695856=793323,2171784=534399,3362453=133403} 2024-04-24T01:56:54.551 INFO:tasks.rados.rados.0.smithi079.stdout:399: writing smithi07929762-48 from 695856 to 1489179 tid 1 2024-04-24T01:56:54.558 INFO:tasks.rados.rados.0.smithi079.stdout:399: writing smithi07929762-48 from 2171784 to 2706183 tid 2 2024-04-24T01:56:54.563 INFO:tasks.rados.rados.0.smithi079.stdout:399: writing smithi07929762-48 from 3362453 to 3495856 tid 3 2024-04-24T01:56:54.563 INFO:tasks.rados.rados.0.smithi079.stdout:394: done (2 left) 2024-04-24T01:56:54.563 INFO:tasks.rados.rados.0.smithi079.stdout:400: write oid 32 current snap is 44 2024-04-24T01:56:54.563 INFO:tasks.rados.rados.0.smithi079.stdout:397: finishing write tid 3 to smithi07929762-23 2024-04-24T01:56:54.563 INFO:tasks.rados.rados.0.smithi079.stdout:397: finishing write tid 4 to smithi07929762-23 2024-04-24T01:56:54.563 INFO:tasks.rados.rados.0.smithi079.stdout:400: seq_num 209 ranges {712663=468572,1713913=678057,2873523=591733} 2024-04-24T01:56:54.569 INFO:tasks.rados.rados.0.smithi079.stdout:400: writing smithi07929762-32 from 712663 to 1181235 tid 1 2024-04-24T01:56:54.577 INFO:tasks.rados.rados.0.smithi079.stdout:400: writing smithi07929762-32 from 1713913 to 2391970 tid 2 2024-04-24T01:56:54.583 INFO:tasks.rados.rados.0.smithi079.stdout:400: writing smithi07929762-32 from 2873523 to 3465256 tid 3 2024-04-24T01:56:54.583 INFO:tasks.rados.rados.0.smithi079.stdout:401: read oid 21 snap -1 2024-04-24T01:56:54.583 INFO:tasks.rados.rados.0.smithi079.stdout:401: expect (ObjNum 20 snap 0 seq_num 20) 2024-04-24T01:56:54.584 INFO:tasks.rados.rados.0.smithi079.stdout:402: snap_create 2024-04-24T01:56:54.584 INFO:tasks.rados.rados.0.smithi079.stdout:397: finishing write tid 5 to smithi07929762-23 2024-04-24T01:56:54.584 INFO:tasks.rados.rados.0.smithi079.stdout:397: oid 23 updating version 0 to 32 2024-04-24T01:56:54.584 INFO:tasks.rados.rados.0.smithi079.stdout:397: oid 23 version 32 is already newer than 29 2024-04-24T01:56:54.584 INFO:tasks.rados.rados.0.smithi079.stdout:397: oid 23 version 32 is already newer than 32 2024-04-24T01:56:54.585 INFO:tasks.rados.rados.0.smithi079.stdout:397: oid 23 version 32 is already newer than 32 2024-04-24T01:56:54.585 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 23 v 32 (ObjNum 206 snap 44 seq_num 206) dirty exists 2024-04-24T01:56:54.585 INFO:tasks.rados.rados.0.smithi079.stdout:397: left oid 23 (ObjNum 206 snap 44 seq_num 206) 2024-04-24T01:56:54.602 INFO:tasks.rados.rados.0.smithi079.stdout:401: expect (ObjNum 20 snap 0 seq_num 20) 2024-04-24T01:56:54.703 INFO:tasks.rados.rados.0.smithi079.stdout:399: finishing write tid 1 to smithi07929762-48 2024-04-24T01:56:54.703 INFO:tasks.rados.rados.0.smithi079.stdout:399: finishing write tid 2 to smithi07929762-48 2024-04-24T01:56:54.703 INFO:tasks.rados.rados.0.smithi079.stdout:399: finishing write tid 3 to smithi07929762-48 2024-04-24T01:56:54.703 INFO:tasks.rados.rados.0.smithi079.stdout:399: finishing write tid 4 to smithi07929762-48 2024-04-24T01:56:54.704 INFO:tasks.rados.rados.0.smithi079.stdout:399: finishing write tid 5 to smithi07929762-48 2024-04-24T01:56:54.704 INFO:tasks.rados.rados.0.smithi079.stdout:399: oid 48 updating version 0 to 37 2024-04-24T01:56:54.704 INFO:tasks.rados.rados.0.smithi079.stdout:399: oid 48 version 37 is already newer than 37 2024-04-24T01:56:54.704 INFO:tasks.rados.rados.0.smithi079.stdout:399: oid 48 version 37 is already newer than 37 2024-04-24T01:56:54.704 INFO:tasks.rados.rados.0.smithi079.stdout:399: oid 48 version 37 is already newer than 37 2024-04-24T01:56:54.704 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 48 v 37 (ObjNum 208 snap 44 seq_num 208) dirty exists 2024-04-24T01:56:54.705 INFO:tasks.rados.rados.0.smithi079.stdout:399: left oid 48 (ObjNum 208 snap 44 seq_num 208) 2024-04-24T01:56:54.705 INFO:tasks.rados.rados.0.smithi079.stdout:400: finishing write tid 1 to smithi07929762-32 2024-04-24T01:56:54.705 INFO:tasks.rados.rados.0.smithi079.stdout:400: finishing write tid 2 to smithi07929762-32 2024-04-24T01:56:54.705 INFO:tasks.rados.rados.0.smithi079.stdout:400: finishing write tid 3 to smithi07929762-32 2024-04-24T01:56:54.705 INFO:tasks.rados.rados.0.smithi079.stdout:400: finishing write tid 4 to smithi07929762-32 2024-04-24T01:56:54.705 INFO:tasks.rados.rados.0.smithi079.stdout:400: finishing write tid 5 to smithi07929762-32 2024-04-24T01:56:54.705 INFO:tasks.rados.rados.0.smithi079.stdout:400: oid 32 updating version 0 to 42 2024-04-24T01:56:54.706 INFO:tasks.rados.rados.0.smithi079.stdout:400: oid 32 version 42 is already newer than 42 2024-04-24T01:56:54.706 INFO:tasks.rados.rados.0.smithi079.stdout:400: oid 32 version 42 is already newer than 42 2024-04-24T01:56:54.706 INFO:tasks.rados.rados.0.smithi079.stdout:400: oid 32 version 42 is already newer than 42 2024-04-24T01:56:54.706 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 32 v 42 (ObjNum 209 snap 44 seq_num 209) dirty exists 2024-04-24T01:56:54.706 INFO:tasks.rados.rados.0.smithi079.stdout:400: left oid 32 (ObjNum 209 snap 44 seq_num 209) 2024-04-24T01:56:55.063 INFO:tasks.rados.rados.0.smithi079.stdout:397: done (4 left) 2024-04-24T01:56:55.063 INFO:tasks.rados.rados.0.smithi079.stdout:399: done (3 left) 2024-04-24T01:56:55.063 INFO:tasks.rados.rados.0.smithi079.stdout:400: done (2 left) 2024-04-24T01:56:55.063 INFO:tasks.rados.rados.0.smithi079.stdout:401: done (1 left) 2024-04-24T01:56:55.064 INFO:tasks.rados.rados.0.smithi079.stdout:402: done (0 left) 2024-04-24T01:56:55.064 INFO:tasks.rados.rados.0.smithi079.stdout:403: read oid 17 snap -1 2024-04-24T01:56:55.064 INFO:tasks.rados.rados.0.smithi079.stdout:403: expect (ObjNum 202 snap 43 seq_num 202) 2024-04-24T01:56:55.064 INFO:tasks.rados.rados.0.smithi079.stdout:404: read oid 35 snap 37 2024-04-24T01:56:55.064 INFO:tasks.rados.rados.0.smithi079.stdout:404: expect (ObjNum 188 snap 36 seq_num 188) 2024-04-24T01:56:55.064 INFO:tasks.rados.rados.0.smithi079.stdout:405: read oid 8 snap -1 2024-04-24T01:56:55.064 INFO:tasks.rados.rados.0.smithi079.stdout:405: expect deleted 2024-04-24T01:56:55.064 INFO:tasks.rados.rados.0.smithi079.stdout:406: snap_remove snap 44 2024-04-24T01:56:55.250 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T01:56:55.356 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:55.353+0000 7fc994ead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:55.359 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:55.359+0000 7fc994ead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:56.025 INFO:tasks.rados.rados.0.smithi079.stdout:406: done (3 left) 2024-04-24T01:56:56.025 INFO:tasks.rados.rados.0.smithi079.stdout:407: delete oid 4 current snap is 45 2024-04-24T01:56:56.025 INFO:tasks.rados.rados.0.smithi079.stdout:403: expect (ObjNum 202 snap 43 seq_num 202) 2024-04-24T01:56:56.256 INFO:tasks.rados.rados.0.smithi079.stdout:404: expect (ObjNum 188 snap 36 seq_num 188) 2024-04-24T01:56:56.482 INFO:tasks.rados.rados.0.smithi079.stdout:403: done (3 left) 2024-04-24T01:56:56.482 INFO:tasks.rados.rados.0.smithi079.stdout:404: done (2 left) 2024-04-24T01:56:56.482 INFO:tasks.rados.rados.0.smithi079.stdout:405: done (1 left) 2024-04-24T01:56:56.482 INFO:tasks.rados.rados.0.smithi079.stdout:407: done (0 left) 2024-04-24T01:56:56.482 INFO:tasks.rados.rados.0.smithi079.stdout:408: write oid 13 current snap is 45 2024-04-24T01:56:56.482 INFO:tasks.rados.rados.0.smithi079.stdout:408: seq_num 211 ranges {746277=619070,1830877=681726,2928369=617908} 2024-04-24T01:56:56.489 INFO:tasks.rados.rados.0.smithi079.stdout:408: writing smithi07929762-13 from 746277 to 1365347 tid 1 2024-04-24T01:56:56.499 INFO:tasks.rados.rados.0.smithi079.stdout:408: writing smithi07929762-13 from 1830877 to 2512603 tid 2 2024-04-24T01:56:56.507 INFO:tasks.rados.rados.0.smithi079.stdout:408: writing smithi07929762-13 from 2928369 to 3546277 tid 3 2024-04-24T01:56:56.507 INFO:tasks.rados.rados.0.smithi079.stdout:409: write (excl) oid 14 current snap is 45 2024-04-24T01:56:56.507 INFO:tasks.rados.rados.0.smithi079.stdout:409: seq_num 212 ranges {763084=694319,1889359=541737,2955792=541902} 2024-04-24T01:56:56.518 INFO:tasks.rados.rados.0.smithi079.stdout:409: writing smithi07929762-14 from 763084 to 1457403 tid 1 2024-04-24T01:56:56.524 INFO:tasks.rados.rados.0.smithi079.stdout:409: writing smithi07929762-14 from 1889359 to 2431096 tid 2 2024-04-24T01:56:56.533 INFO:tasks.rados.rados.0.smithi079.stdout:409: writing smithi07929762-14 from 2955792 to 3497694 tid 3 2024-04-24T01:56:56.533 INFO:tasks.rados.rados.0.smithi079.stdout:410: read oid 27 snap -1 2024-04-24T01:56:56.533 INFO:tasks.rados.rados.0.smithi079.stdout:410: expect deleted 2024-04-24T01:56:56.533 INFO:tasks.rados.rados.0.smithi079.stdout:411: snap_remove snap 37 2024-04-24T01:56:57.069 INFO:tasks.rados.rados.0.smithi079.stdout:411: done (3 left) 2024-04-24T01:56:57.069 INFO:tasks.rados.rados.0.smithi079.stdout:412: write (excl) oid 31 current snap is 45 2024-04-24T01:56:57.069 INFO:tasks.rados.rados.0.smithi079.stdout:412: seq_num 213 ranges {779891=485921,1947841=685395,3383215=196676} 2024-04-24T01:56:57.076 INFO:tasks.rados.rados.0.smithi079.stdout:412: writing smithi07929762-31 from 779891 to 1265812 tid 1 2024-04-24T01:56:57.083 INFO:tasks.rados.rados.0.smithi079.stdout:412: writing smithi07929762-31 from 1947841 to 2633236 tid 2 2024-04-24T01:56:57.089 INFO:tasks.rados.rados.0.smithi079.stdout:412: writing smithi07929762-31 from 3383215 to 3579891 tid 3 2024-04-24T01:56:57.089 INFO:tasks.rados.rados.0.smithi079.stdout:413: snap_create 2024-04-24T01:56:57.090 INFO:tasks.rados.rados.0.smithi079.stdout:409: finishing write tid 1 to smithi07929762-14 2024-04-24T01:56:57.090 INFO:tasks.rados.rados.0.smithi079.stdout:408: finishing write tid 1 to smithi07929762-13 2024-04-24T01:56:57.090 INFO:tasks.rados.rados.0.smithi079.stdout:409: finishing write tid 2 to smithi07929762-14 2024-04-24T01:56:57.090 INFO:tasks.rados.rados.0.smithi079.stdout:408: finishing write tid 2 to smithi07929762-13 2024-04-24T01:56:57.090 INFO:tasks.rados.rados.0.smithi079.stdout:409: finishing write tid 3 to smithi07929762-14 2024-04-24T01:56:57.090 INFO:tasks.rados.rados.0.smithi079.stdout:408: finishing write tid 3 to smithi07929762-13 2024-04-24T01:56:57.090 INFO:tasks.rados.rados.0.smithi079.stdout:409: finishing write tid 4 to smithi07929762-14 2024-04-24T01:56:57.090 INFO:tasks.rados.rados.0.smithi079.stdout:409: finishing write tid 5 to smithi07929762-14 2024-04-24T01:56:57.090 INFO:tasks.rados.rados.0.smithi079.stdout:409: oid 14 updating version 0 to 83 2024-04-24T01:56:57.090 INFO:tasks.rados.rados.0.smithi079.stdout:409: oid 14 version 83 is already newer than 80 2024-04-24T01:56:57.090 INFO:tasks.rados.rados.0.smithi079.stdout:409: oid 14 version 83 is already newer than 83 2024-04-24T01:56:57.090 INFO:tasks.rados.rados.0.smithi079.stdout:409: oid 14 version 83 is already newer than 83 2024-04-24T01:56:57.091 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 14 v 83 (ObjNum 212 snap 45 seq_num 212) dirty exists 2024-04-24T01:56:57.091 INFO:tasks.rados.rados.0.smithi079.stdout:409: left oid 14 (ObjNum 212 snap 45 seq_num 212) 2024-04-24T01:56:57.091 INFO:tasks.rados.rados.0.smithi079.stdout:408: finishing write tid 4 to smithi07929762-13 2024-04-24T01:56:57.091 INFO:tasks.rados.rados.0.smithi079.stdout:408: finishing write tid 5 to smithi07929762-13 2024-04-24T01:56:57.091 INFO:tasks.rados.rados.0.smithi079.stdout:408: oid 13 updating version 0 to 95 2024-04-24T01:56:57.091 INFO:tasks.rados.rados.0.smithi079.stdout:408: oid 13 version 95 is already newer than 95 2024-04-24T01:56:57.091 INFO:tasks.rados.rados.0.smithi079.stdout:408: oid 13 version 95 is already newer than 95 2024-04-24T01:56:57.091 INFO:tasks.rados.rados.0.smithi079.stdout:408: oid 13 version 95 is already newer than 95 2024-04-24T01:56:57.091 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 13 v 95 (ObjNum 211 snap 45 seq_num 211) dirty exists 2024-04-24T01:56:57.091 INFO:tasks.rados.rados.0.smithi079.stdout:408: left oid 13 (ObjNum 211 snap 45 seq_num 211) 2024-04-24T01:56:57.162 INFO:tasks.rados.rados.0.smithi079.stdout:412: finishing write tid 1 to smithi07929762-31 2024-04-24T01:56:57.181 INFO:tasks.rados.rados.0.smithi079.stdout:412: finishing write tid 2 to smithi07929762-31 2024-04-24T01:56:57.185 INFO:tasks.rados.rados.0.smithi079.stdout:412: finishing write tid 3 to smithi07929762-31 2024-04-24T01:56:57.188 INFO:tasks.rados.rados.0.smithi079.stdout:412: finishing write tid 4 to smithi07929762-31 2024-04-24T01:56:57.189 INFO:tasks.rados.rados.0.smithi079.stdout:412: finishing write tid 5 to smithi07929762-31 2024-04-24T01:56:57.190 INFO:tasks.rados.rados.0.smithi079.stdout:412: oid 31 updating version 0 to 127 2024-04-24T01:56:57.190 INFO:tasks.rados.rados.0.smithi079.stdout:412: oid 31 version 127 is already newer than 127 2024-04-24T01:56:57.190 INFO:tasks.rados.rados.0.smithi079.stdout:412: oid 31 version 127 is already newer than 127 2024-04-24T01:56:57.190 INFO:tasks.rados.rados.0.smithi079.stdout:412: oid 31 version 127 is already newer than 124 2024-04-24T01:56:57.190 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 31 v 127 (ObjNum 213 snap 45 seq_num 213) dirty exists 2024-04-24T01:56:57.190 INFO:tasks.rados.rados.0.smithi079.stdout:412: left oid 31 (ObjNum 213 snap 45 seq_num 213) 2024-04-24T01:56:57.452 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T01:56:57.557 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:57.555+0000 7f0da2bff640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:57.560 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:57.560+0000 7f0da199d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:58.069 INFO:tasks.rados.rados.0.smithi079.stdout:408: done (4 left) 2024-04-24T01:56:58.069 INFO:tasks.rados.rados.0.smithi079.stdout:409: done (3 left) 2024-04-24T01:56:58.069 INFO:tasks.rados.rados.0.smithi079.stdout:410: done (2 left) 2024-04-24T01:56:58.069 INFO:tasks.rados.rados.0.smithi079.stdout:412: done (1 left) 2024-04-24T01:56:58.069 INFO:tasks.rados.rados.0.smithi079.stdout:413: done (0 left) 2024-04-24T01:56:58.069 INFO:tasks.rados.rados.0.smithi079.stdout:414: snap_remove snap 43 2024-04-24T01:56:58.365 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-04-24T01:56:58.366 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-04-24T01:56:58.366 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-04-24T01:56:58.366 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-04-24T01:56:58.488 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:58.483+0000 7f3f58aad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:58.492 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:58.491+0000 7f3f58aad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:58.506 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T01:56:58.506 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T01:56:58.506 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T01:56:58.506 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T01:56:58.507 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T01:56:58.507 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T01:56:58.507 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T01:56:58.507 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T01:56:58.507 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T01:56:58.507 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T01:56:58.508 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T01:56:58.508 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T01:56:58.508 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T01:56:58.509 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T01:56:58.509 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-04-24T01:56:58.617 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:58.612+0000 7f6d44ced640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:58.621 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:58.621+0000 7f6d44ced640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:58.635 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T01:56:58.636 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T01:56:58.636 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T01:56:58.636 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T01:56:58.636 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T01:56:58.636 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T01:56:58.636 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T01:56:58.636 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T01:56:58.637 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T01:56:58.637 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T01:56:58.637 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T01:56:58.637 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T01:56:58.639 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T01:56:58.639 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T01:56:58.639 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-04-24T01:56:58.747 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:58.742+0000 7fd1d391d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:58.750 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:58.750+0000 7fd1d391d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:58.766 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T01:56:58.766 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T01:56:58.766 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T01:56:58.766 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T01:56:58.766 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T01:56:58.767 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T01:56:58.767 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T01:56:58.767 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T01:56:58.767 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T01:56:58.767 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T01:56:58.767 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T01:56:58.767 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T01:56:58.769 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T01:56:58.769 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T01:56:58.770 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-04-24T01:56:58.876 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:58.872+0000 7fe2f126d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:58.880 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:58.880+0000 7fe2f126d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:58.896 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T01:56:58.896 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T01:56:58.896 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T01:56:58.896 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T01:56:58.896 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T01:56:58.896 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T01:56:58.896 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T01:56:58.896 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T01:56:58.896 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T01:56:58.896 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T01:56:58.896 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T01:56:58.897 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T01:56:58.899 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T01:56:58.899 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T01:56:59.072 INFO:tasks.rados.rados.0.smithi079.stdout:414: done (0 left) 2024-04-24T01:56:59.073 INFO:tasks.rados.rados.0.smithi079.stdout:415: read oid 22 snap -1 2024-04-24T01:56:59.073 INFO:tasks.rados.rados.0.smithi079.stdout:415: expect (ObjNum 195 snap 40 seq_num 195) 2024-04-24T01:56:59.073 INFO:tasks.rados.rados.0.smithi079.stdout:416: snap_remove snap 38 2024-04-24T01:56:59.653 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T01:56:59.761 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:59.760+0000 7fb5b039d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:56:59.765 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:56:59.765+0000 7fb5b039d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:00.073 INFO:tasks.rados.rados.0.smithi079.stdout:416: done (1 left) 2024-04-24T01:57:00.074 INFO:tasks.rados.rados.0.smithi079.stdout:417: write (excl) oid 14 current snap is 46 2024-04-24T01:57:00.074 INFO:tasks.rados.rados.0.smithi079.stdout:417: seq_num 214 ranges {796698=561170,2006323=545406,3126991=469707} 2024-04-24T01:57:00.081 INFO:tasks.rados.rados.0.smithi079.stdout:417: writing smithi07929762-14 from 796698 to 1357868 tid 1 2024-04-24T01:57:00.088 INFO:tasks.rados.rados.0.smithi079.stdout:417: writing smithi07929762-14 from 2006323 to 2551729 tid 2 2024-04-24T01:57:00.094 INFO:tasks.rados.rados.0.smithi079.stdout:417: writing smithi07929762-14 from 3126991 to 3596698 tid 3 2024-04-24T01:57:00.094 INFO:tasks.rados.rados.0.smithi079.stdout:418: write oid 11 current snap is 46 2024-04-24T01:57:00.094 INFO:tasks.rados.rados.0.smithi079.stdout:418: seq_num 215 ranges {413505=636419,1664805=689064,3038061=492071} 2024-04-24T01:57:00.100 INFO:tasks.rados.rados.0.smithi079.stdout:418: writing smithi07929762-11 from 413505 to 1049924 tid 1 2024-04-24T01:57:00.107 INFO:tasks.rados.rados.0.smithi079.stdout:418: writing smithi07929762-11 from 1664805 to 2353869 tid 2 2024-04-24T01:57:00.115 INFO:tasks.rados.rados.0.smithi079.stdout:418: writing smithi07929762-11 from 3038061 to 3530132 tid 3 2024-04-24T01:57:00.115 INFO:tasks.rados.rados.0.smithi079.stdout:419: read oid 30 snap -1 2024-04-24T01:57:00.115 INFO:tasks.rados.rados.0.smithi079.stdout:419: expect (ObjNum 150 snap 20 seq_num 150) 2024-04-24T01:57:00.115 INFO:tasks.rados.rados.0.smithi079.stdout:420: snap_create 2024-04-24T01:57:00.115 INFO:tasks.rados.rados.0.smithi079.stdout:415: expect (ObjNum 195 snap 40 seq_num 195) 2024-04-24T01:57:00.277 INFO:tasks.rados.rados.0.smithi079.stdout:419: expect (ObjNum 150 snap 20 seq_num 150) 2024-04-24T01:57:00.384 INFO:tasks.rados.rados.0.smithi079.stdout:417: finishing write tid 1 to smithi07929762-14 2024-04-24T01:57:00.385 INFO:tasks.rados.rados.0.smithi079.stdout:417: finishing write tid 2 to smithi07929762-14 2024-04-24T01:57:00.385 INFO:tasks.rados.rados.0.smithi079.stdout:417: finishing write tid 3 to smithi07929762-14 2024-04-24T01:57:00.385 INFO:tasks.rados.rados.0.smithi079.stdout:417: finishing write tid 4 to smithi07929762-14 2024-04-24T01:57:00.385 INFO:tasks.rados.rados.0.smithi079.stdout:417: finishing write tid 5 to smithi07929762-14 2024-04-24T01:57:00.385 INFO:tasks.rados.rados.0.smithi079.stdout:417: oid 14 updating version 0 to 96 2024-04-24T01:57:00.385 INFO:tasks.rados.rados.0.smithi079.stdout:417: oid 14 version 96 is already newer than 93 2024-04-24T01:57:00.385 INFO:tasks.rados.rados.0.smithi079.stdout:417: oid 14 version 96 is already newer than 96 2024-04-24T01:57:00.385 INFO:tasks.rados.rados.0.smithi079.stdout:417: oid 14 version 96 is already newer than 96 2024-04-24T01:57:00.385 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 14 v 96 (ObjNum 214 snap 46 seq_num 214) dirty exists 2024-04-24T01:57:00.386 INFO:tasks.rados.rados.0.smithi079.stdout:417: left oid 14 (ObjNum 214 snap 46 seq_num 214) 2024-04-24T01:57:00.386 INFO:tasks.rados.rados.0.smithi079.stdout:418: finishing write tid 1 to smithi07929762-11 2024-04-24T01:57:00.386 INFO:tasks.rados.rados.0.smithi079.stdout:418: finishing write tid 2 to smithi07929762-11 2024-04-24T01:57:00.386 INFO:tasks.rados.rados.0.smithi079.stdout:418: finishing write tid 3 to smithi07929762-11 2024-04-24T01:57:00.386 INFO:tasks.rados.rados.0.smithi079.stdout:418: finishing write tid 4 to smithi07929762-11 2024-04-24T01:57:00.386 INFO:tasks.rados.rados.0.smithi079.stdout:418: finishing write tid 5 to smithi07929762-11 2024-04-24T01:57:00.386 INFO:tasks.rados.rados.0.smithi079.stdout:418: oid 11 updating version 0 to 100 2024-04-24T01:57:00.386 INFO:tasks.rados.rados.0.smithi079.stdout:418: oid 11 version 100 is already newer than 100 2024-04-24T01:57:00.386 INFO:tasks.rados.rados.0.smithi079.stdout:418: oid 11 version 100 is already newer than 100 2024-04-24T01:57:00.386 INFO:tasks.rados.rados.0.smithi079.stdout:418: oid 11 version 100 is already newer than 100 2024-04-24T01:57:00.386 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 11 v 100 (ObjNum 215 snap 46 seq_num 215) dirty exists 2024-04-24T01:57:00.386 INFO:tasks.rados.rados.0.smithi079.stdout:418: left oid 11 (ObjNum 215 snap 46 seq_num 215) 2024-04-24T01:57:01.082 INFO:tasks.rados.rados.0.smithi079.stdout:415: done (4 left) 2024-04-24T01:57:01.082 INFO:tasks.rados.rados.0.smithi079.stdout:417: done (3 left) 2024-04-24T01:57:01.082 INFO:tasks.rados.rados.0.smithi079.stdout:418: done (2 left) 2024-04-24T01:57:01.082 INFO:tasks.rados.rados.0.smithi079.stdout:419: done (1 left) 2024-04-24T01:57:01.082 INFO:tasks.rados.rados.0.smithi079.stdout:420: done (0 left) 2024-04-24T01:57:01.082 INFO:tasks.rados.rados.0.smithi079.stdout:421: write (excl) oid 13 current snap is 47 2024-04-24T01:57:01.082 INFO:tasks.rados.rados.0.smithi079.stdout:421: seq_num 216 ranges {430312=711668,1723287=549075,2781837=703343} 2024-04-24T01:57:01.086 INFO:tasks.rados.rados.0.smithi079.stdout:421: writing smithi07929762-13 from 430312 to 1141980 tid 1 2024-04-24T01:57:01.092 INFO:tasks.rados.rados.0.smithi079.stdout:421: writing smithi07929762-13 from 1723287 to 2272362 tid 2 2024-04-24T01:57:01.099 INFO:tasks.rados.rados.0.smithi079.stdout:421: writing smithi07929762-13 from 2781837 to 3485180 tid 3 2024-04-24T01:57:01.099 INFO:tasks.rados.rados.0.smithi079.stdout:422: write oid 48 current snap is 47 2024-04-24T01:57:01.099 INFO:tasks.rados.rados.0.smithi079.stdout:422: seq_num 217 ranges {447119=786917,1665416=692733,2976554=514615} 2024-04-24T01:57:01.106 INFO:tasks.rados.rados.0.smithi079.stdout:422: writing smithi07929762-48 from 447119 to 1234036 tid 1 2024-04-24T01:57:01.113 INFO:tasks.rados.rados.0.smithi079.stdout:422: writing smithi07929762-48 from 1665416 to 2358149 tid 2 2024-04-24T01:57:01.120 INFO:tasks.rados.rados.0.smithi079.stdout:422: writing smithi07929762-48 from 2976554 to 3491169 tid 3 2024-04-24T01:57:01.120 INFO:tasks.rados.rados.0.smithi079.stdout:423: write (excl) oid 12 current snap is 47 2024-04-24T01:57:01.120 INFO:tasks.rados.rados.0.smithi079.stdout:423: seq_num 218 ranges {463926=462166,1723898=436391,2603977=725887} 2024-04-24T01:57:01.125 INFO:tasks.rados.rados.0.smithi079.stdout:423: writing smithi07929762-12 from 463926 to 926092 tid 1 2024-04-24T01:57:01.132 INFO:tasks.rados.rados.0.smithi079.stdout:423: writing smithi07929762-12 from 1723898 to 2160289 tid 2 2024-04-24T01:57:01.141 INFO:tasks.rados.rados.0.smithi079.stdout:423: writing smithi07929762-12 from 2603977 to 3329864 tid 3 2024-04-24T01:57:01.141 INFO:tasks.rados.rados.0.smithi079.stdout:424: snap_remove snap 45 2024-04-24T01:57:01.858 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T01:57:01.970 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:01.969+0000 7f4e0240f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:01.974 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:01.974+0000 7f4e0240f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:02.083 INFO:tasks.rados.rados.0.smithi079.stdout:424: done (3 left) 2024-04-24T01:57:02.083 INFO:tasks.rados.rados.0.smithi079.stdout:425: snap_create 2024-04-24T01:57:02.083 INFO:tasks.rados.rados.0.smithi079.stdout:423: finishing write tid 1 to smithi07929762-12 2024-04-24T01:57:02.083 INFO:tasks.rados.rados.0.smithi079.stdout:421: finishing write tid 1 to smithi07929762-13 2024-04-24T01:57:02.083 INFO:tasks.rados.rados.0.smithi079.stdout:422: finishing write tid 1 to smithi07929762-48 2024-04-24T01:57:02.083 INFO:tasks.rados.rados.0.smithi079.stdout:423: finishing write tid 2 to smithi07929762-12 2024-04-24T01:57:02.083 INFO:tasks.rados.rados.0.smithi079.stdout:421: finishing write tid 2 to smithi07929762-13 2024-04-24T01:57:02.083 INFO:tasks.rados.rados.0.smithi079.stdout:423: finishing write tid 3 to smithi07929762-12 2024-04-24T01:57:02.083 INFO:tasks.rados.rados.0.smithi079.stdout:422: finishing write tid 2 to smithi07929762-48 2024-04-24T01:57:02.084 INFO:tasks.rados.rados.0.smithi079.stdout:421: finishing write tid 3 to smithi07929762-13 2024-04-24T01:57:02.084 INFO:tasks.rados.rados.0.smithi079.stdout:423: finishing write tid 4 to smithi07929762-12 2024-04-24T01:57:02.084 INFO:tasks.rados.rados.0.smithi079.stdout:423: finishing write tid 5 to smithi07929762-12 2024-04-24T01:57:02.084 INFO:tasks.rados.rados.0.smithi079.stdout:423: oid 12 updating version 0 to 105 2024-04-24T01:57:02.084 INFO:tasks.rados.rados.0.smithi079.stdout:423: oid 12 version 105 is already newer than 105 2024-04-24T01:57:02.084 INFO:tasks.rados.rados.0.smithi079.stdout:423: oid 12 version 105 is already newer than 105 2024-04-24T01:57:02.084 INFO:tasks.rados.rados.0.smithi079.stdout:423: oid 12 version 105 is already newer than 102 2024-04-24T01:57:02.084 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 12 v 105 (ObjNum 218 snap 47 seq_num 218) dirty exists 2024-04-24T01:57:02.084 INFO:tasks.rados.rados.0.smithi079.stdout:423: left oid 12 (ObjNum 218 snap 47 seq_num 218) 2024-04-24T01:57:02.084 INFO:tasks.rados.rados.0.smithi079.stdout:421: finishing write tid 4 to smithi07929762-13 2024-04-24T01:57:02.084 INFO:tasks.rados.rados.0.smithi079.stdout:422: finishing write tid 3 to smithi07929762-48 2024-04-24T01:57:02.084 INFO:tasks.rados.rados.0.smithi079.stdout:421: finishing write tid 5 to smithi07929762-13 2024-04-24T01:57:02.084 INFO:tasks.rados.rados.0.smithi079.stdout:421: oid 13 updating version 0 to 106 2024-04-24T01:57:02.084 INFO:tasks.rados.rados.0.smithi079.stdout:421: oid 13 version 106 is already newer than 106 2024-04-24T01:57:02.084 INFO:tasks.rados.rados.0.smithi079.stdout:421: oid 13 version 106 is already newer than 106 2024-04-24T01:57:02.085 INFO:tasks.rados.rados.0.smithi079.stdout:421: oid 13 version 106 is already newer than 103 2024-04-24T01:57:02.085 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 13 v 106 (ObjNum 216 snap 47 seq_num 216) dirty exists 2024-04-24T01:57:02.085 INFO:tasks.rados.rados.0.smithi079.stdout:421: left oid 13 (ObjNum 216 snap 47 seq_num 216) 2024-04-24T01:57:02.085 INFO:tasks.rados.rados.0.smithi079.stdout:422: finishing write tid 4 to smithi07929762-48 2024-04-24T01:57:02.085 INFO:tasks.rados.rados.0.smithi079.stdout:422: finishing write tid 5 to smithi07929762-48 2024-04-24T01:57:02.085 INFO:tasks.rados.rados.0.smithi079.stdout:422: oid 48 updating version 0 to 55 2024-04-24T01:57:02.085 INFO:tasks.rados.rados.0.smithi079.stdout:422: oid 48 version 55 is already newer than 55 2024-04-24T01:57:02.085 INFO:tasks.rados.rados.0.smithi079.stdout:422: oid 48 version 55 is already newer than 55 2024-04-24T01:57:02.085 INFO:tasks.rados.rados.0.smithi079.stdout:422: oid 48 version 55 is already newer than 55 2024-04-24T01:57:02.085 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 48 v 55 (ObjNum 217 snap 47 seq_num 217) dirty exists 2024-04-24T01:57:02.085 INFO:tasks.rados.rados.0.smithi079.stdout:422: left oid 48 (ObjNum 217 snap 47 seq_num 217) 2024-04-24T01:57:03.088 INFO:tasks.rados.rados.0.smithi079.stdout:421: done (3 left) 2024-04-24T01:57:03.088 INFO:tasks.rados.rados.0.smithi079.stdout:422: done (2 left) 2024-04-24T01:57:03.088 INFO:tasks.rados.rados.0.smithi079.stdout:423: done (1 left) 2024-04-24T01:57:03.088 INFO:tasks.rados.rados.0.smithi079.stdout:425: done (0 left) 2024-04-24T01:57:03.088 INFO:tasks.rados.rados.0.smithi079.stdout:426: snap_create 2024-04-24T01:57:03.900 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-04-24T01:57:03.901 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-04-24T01:57:03.901 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-04-24T01:57:03.901 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-04-24T01:57:04.022 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:04.017+0000 7f6f6d11d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:04.025 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:04.026+0000 7f6f6d11d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:04.040 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T01:57:04.040 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T01:57:04.040 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T01:57:04.041 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T01:57:04.041 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T01:57:04.041 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T01:57:04.041 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T01:57:04.041 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T01:57:04.041 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T01:57:04.041 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T01:57:04.041 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T01:57:04.041 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T01:57:04.043 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T01:57:04.044 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T01:57:04.044 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-04-24T01:57:04.069 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T01:57:04.088 INFO:tasks.rados.rados.0.smithi079.stdout:426: done (0 left) 2024-04-24T01:57:04.088 INFO:tasks.rados.rados.0.smithi079.stdout:427: write (excl) oid 14 current snap is 49 2024-04-24T01:57:04.088 INFO:tasks.rados.rados.0.smithi079.stdout:427: seq_num 219 ranges {480733=537415,1782380=696402,3031400=537159} 2024-04-24T01:57:04.096 INFO:tasks.rados.rados.0.smithi079.stdout:427: writing smithi07929762-14 from 480733 to 1018148 tid 1 2024-04-24T01:57:04.108 INFO:tasks.rados.rados.0.smithi079.stdout:427: writing smithi07929762-14 from 1782380 to 2478782 tid 2 2024-04-24T01:57:04.117 INFO:tasks.rados.rados.0.smithi079.stdout:427: writing smithi07929762-14 from 3031400 to 3568559 tid 3 2024-04-24T01:57:04.117 INFO:tasks.rados.rados.0.smithi079.stdout:428: write (excl) oid 17 current snap is 49 2024-04-24T01:57:04.117 INFO:tasks.rados.rados.0.smithi079.stdout:428: seq_num 220 ranges {497540=612664,1840862=440060,3058823=464784} 2024-04-24T01:57:04.126 INFO:tasks.rados.rados.0.smithi079.stdout:428: writing smithi07929762-17 from 497540 to 1110204 tid 1 2024-04-24T01:57:04.134 INFO:tasks.rados.rados.0.smithi079.stdout:428: writing smithi07929762-17 from 1840862 to 2280922 tid 2 2024-04-24T01:57:04.144 INFO:tasks.rados.rados.0.smithi079.stdout:428: writing smithi07929762-17 from 3058823 to 3523607 tid 3 2024-04-24T01:57:04.144 INFO:tasks.rados.rados.0.smithi079.stdout:429: write oid 24 current snap is 49 2024-04-24T01:57:04.144 INFO:tasks.rados.rados.0.smithi079.stdout:429: seq_num 221 ranges {514347=404266,1499344=700071,2686246=676056} 2024-04-24T01:57:04.151 INFO:tasks.rados.rados.0.smithi079.stdout:429: writing smithi07929762-24 from 514347 to 918613 tid 1 2024-04-24T01:57:04.162 INFO:tasks.rados.rados.0.smithi079.stdout:429: writing smithi07929762-24 from 1499344 to 2199415 tid 2 2024-04-24T01:57:04.171 INFO:tasks.rados.rados.0.smithi079.stdout:429: writing smithi07929762-24 from 2686246 to 3362302 tid 3 2024-04-24T01:57:04.172 INFO:tasks.rados.rados.0.smithi079.stdout:430: snap_remove snap 41 2024-04-24T01:57:04.190 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:04.180+0000 7f019602d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:04.195 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:04.196+0000 7f019602d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:04.216 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T01:57:04.217 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T01:57:04.217 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T01:57:04.217 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T01:57:04.217 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T01:57:04.217 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T01:57:04.217 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T01:57:04.217 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T01:57:04.217 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T01:57:04.217 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T01:57:04.217 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T01:57:04.217 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T01:57:04.219 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:04.217+0000 7fc995e64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:04.220 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T01:57:04.220 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T01:57:04.221 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-04-24T01:57:04.223 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:04.223+0000 7fc995e64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:04.345 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:04.337+0000 7fcb0046d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:04.349 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:04.349+0000 7fcb0046d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:04.376 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T01:57:04.376 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T01:57:04.376 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T01:57:04.376 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T01:57:04.376 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T01:57:04.376 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T01:57:04.376 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T01:57:04.376 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T01:57:04.376 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T01:57:04.376 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T01:57:04.376 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T01:57:04.377 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T01:57:04.378 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T01:57:04.379 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T01:57:04.379 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-04-24T01:57:04.487 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:04.482+0000 7f560e0ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:04.491 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:04.491+0000 7f560e0ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:04.507 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T01:57:04.507 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T01:57:04.507 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T01:57:04.507 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T01:57:04.507 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T01:57:04.508 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T01:57:04.508 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T01:57:04.508 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T01:57:04.508 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T01:57:04.508 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T01:57:04.508 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T01:57:04.508 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T01:57:04.510 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T01:57:04.511 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T01:57:05.103 INFO:tasks.rados.rados.0.smithi079.stdout:430: done (3 left) 2024-04-24T01:57:05.103 INFO:tasks.rados.rados.0.smithi079.stdout:431: snap_create 2024-04-24T01:57:05.103 INFO:tasks.rados.rados.0.smithi079.stdout:428: finishing write tid 1 to smithi07929762-17 2024-04-24T01:57:05.103 INFO:tasks.rados.rados.0.smithi079.stdout:427: finishing write tid 1 to smithi07929762-14 2024-04-24T01:57:05.103 INFO:tasks.rados.rados.0.smithi079.stdout:428: finishing write tid 2 to smithi07929762-17 2024-04-24T01:57:05.103 INFO:tasks.rados.rados.0.smithi079.stdout:428: finishing write tid 3 to smithi07929762-17 2024-04-24T01:57:05.103 INFO:tasks.rados.rados.0.smithi079.stdout:428: finishing write tid 4 to smithi07929762-17 2024-04-24T01:57:05.103 INFO:tasks.rados.rados.0.smithi079.stdout:428: finishing write tid 5 to smithi07929762-17 2024-04-24T01:57:05.103 INFO:tasks.rados.rados.0.smithi079.stdout:428: oid 17 updating version 0 to 109 2024-04-24T01:57:05.104 INFO:tasks.rados.rados.0.smithi079.stdout:428: oid 17 updating version 109 to 112 2024-04-24T01:57:05.104 INFO:tasks.rados.rados.0.smithi079.stdout:428: oid 17 version 112 is already newer than 112 2024-04-24T01:57:05.104 INFO:tasks.rados.rados.0.smithi079.stdout:428: oid 17 version 112 is already newer than 112 2024-04-24T01:57:05.104 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 17 v 112 (ObjNum 220 snap 49 seq_num 220) dirty exists 2024-04-24T01:57:05.104 INFO:tasks.rados.rados.0.smithi079.stdout:428: left oid 17 (ObjNum 220 snap 49 seq_num 220) 2024-04-24T01:57:05.104 INFO:tasks.rados.rados.0.smithi079.stdout:427: finishing write tid 2 to smithi07929762-14 2024-04-24T01:57:05.104 INFO:tasks.rados.rados.0.smithi079.stdout:429: finishing write tid 1 to smithi07929762-24 2024-04-24T01:57:05.104 INFO:tasks.rados.rados.0.smithi079.stdout:429: finishing write tid 2 to smithi07929762-24 2024-04-24T01:57:05.104 INFO:tasks.rados.rados.0.smithi079.stdout:427: finishing write tid 3 to smithi07929762-14 2024-04-24T01:57:05.104 INFO:tasks.rados.rados.0.smithi079.stdout:427: finishing write tid 4 to smithi07929762-14 2024-04-24T01:57:05.104 INFO:tasks.rados.rados.0.smithi079.stdout:427: finishing write tid 5 to smithi07929762-14 2024-04-24T01:57:05.104 INFO:tasks.rados.rados.0.smithi079.stdout:427: oid 14 updating version 0 to 107 2024-04-24T01:57:05.104 INFO:tasks.rados.rados.0.smithi079.stdout:427: oid 14 version 107 is already newer than 107 2024-04-24T01:57:05.104 INFO:tasks.rados.rados.0.smithi079.stdout:427: oid 14 version 107 is already newer than 104 2024-04-24T01:57:05.104 INFO:tasks.rados.rados.0.smithi079.stdout:427: oid 14 version 107 is already newer than 107 2024-04-24T01:57:05.104 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 14 v 107 (ObjNum 219 snap 49 seq_num 219) dirty exists 2024-04-24T01:57:05.105 INFO:tasks.rados.rados.0.smithi079.stdout:427: left oid 14 (ObjNum 219 snap 49 seq_num 219) 2024-04-24T01:57:05.105 INFO:tasks.rados.rados.0.smithi079.stdout:429: finishing write tid 3 to smithi07929762-24 2024-04-24T01:57:05.105 INFO:tasks.rados.rados.0.smithi079.stdout:429: finishing write tid 4 to smithi07929762-24 2024-04-24T01:57:05.105 INFO:tasks.rados.rados.0.smithi079.stdout:429: finishing write tid 5 to smithi07929762-24 2024-04-24T01:57:05.105 INFO:tasks.rados.rados.0.smithi079.stdout:429: oid 24 updating version 0 to 77 2024-04-24T01:57:05.105 INFO:tasks.rados.rados.0.smithi079.stdout:429: oid 24 version 77 is already newer than 77 2024-04-24T01:57:05.105 INFO:tasks.rados.rados.0.smithi079.stdout:429: oid 24 version 77 is already newer than 77 2024-04-24T01:57:05.105 INFO:tasks.rados.rados.0.smithi079.stdout:429: oid 24 version 77 is already newer than 77 2024-04-24T01:57:05.105 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 24 v 77 (ObjNum 221 snap 49 seq_num 221) dirty exists 2024-04-24T01:57:05.105 INFO:tasks.rados.rados.0.smithi079.stdout:429: left oid 24 (ObjNum 221 snap 49 seq_num 221) 2024-04-24T01:57:06.029 INFO:tasks.rados.rados.0.smithi079.stdout:427: done (3 left) 2024-04-24T01:57:06.029 INFO:tasks.rados.rados.0.smithi079.stdout:428: done (2 left) 2024-04-24T01:57:06.029 INFO:tasks.rados.rados.0.smithi079.stdout:429: done (1 left) 2024-04-24T01:57:06.029 INFO:tasks.rados.rados.0.smithi079.stdout:431: done (0 left) 2024-04-24T01:57:06.029 INFO:tasks.rados.rados.0.smithi079.stdout:432: delete oid 12 current snap is 50 2024-04-24T01:57:06.095 INFO:tasks.rados.rados.0.smithi079.stdout:432: done (0 left) 2024-04-24T01:57:06.095 INFO:tasks.rados.rados.0.smithi079.stdout:433: write (excl) oid 43 current snap is 50 2024-04-24T01:57:06.095 INFO:tasks.rados.rados.0.smithi079.stdout:433: seq_num 223 ranges {547961=554764,1616308=703740,2741092=698600} 2024-04-24T01:57:06.101 INFO:tasks.rados.rados.0.smithi079.stdout:433: writing smithi07929762-43 from 547961 to 1102725 tid 1 2024-04-24T01:57:06.107 INFO:tasks.rados.rados.0.smithi079.stdout:433: writing smithi07929762-43 from 1616308 to 2320048 tid 2 2024-04-24T01:57:06.114 INFO:tasks.rados.rados.0.smithi079.stdout:433: writing smithi07929762-43 from 2741092 to 3439692 tid 3 2024-04-24T01:57:06.114 INFO:tasks.rados.rados.0.smithi079.stdout:434: snap_create 2024-04-24T01:57:06.199 INFO:tasks.rados.rados.0.smithi079.stdout:433: finishing write tid 1 to smithi07929762-43 2024-04-24T01:57:06.237 INFO:tasks.rados.rados.0.smithi079.stdout:433: finishing write tid 2 to smithi07929762-43 2024-04-24T01:57:06.252 INFO:tasks.rados.rados.0.smithi079.stdout:433: finishing write tid 3 to smithi07929762-43 2024-04-24T01:57:06.265 INFO:tasks.rados.rados.0.smithi079.stdout:433: finishing write tid 4 to smithi07929762-43 2024-04-24T01:57:06.267 INFO:tasks.rados.rados.0.smithi079.stdout:433: finishing write tid 5 to smithi07929762-43 2024-04-24T01:57:06.268 INFO:tasks.rados.rados.0.smithi079.stdout:433: oid 43 updating version 0 to 32 2024-04-24T01:57:06.268 INFO:tasks.rados.rados.0.smithi079.stdout:433: oid 43 version 32 is already newer than 29 2024-04-24T01:57:06.268 INFO:tasks.rados.rados.0.smithi079.stdout:433: oid 43 version 32 is already newer than 32 2024-04-24T01:57:06.268 INFO:tasks.rados.rados.0.smithi079.stdout:433: oid 43 version 32 is already newer than 32 2024-04-24T01:57:06.268 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 43 v 32 (ObjNum 223 snap 50 seq_num 223) dirty exists 2024-04-24T01:57:06.268 INFO:tasks.rados.rados.0.smithi079.stdout:433: left oid 43 (ObjNum 223 snap 50 seq_num 223) 2024-04-24T01:57:06.398 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T01:57:06.504 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:06.501+0000 7f1fb962d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:06.508 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:06.507+0000 7f1fb962d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:07.105 INFO:tasks.rados.rados.0.smithi079.stdout:433: done (1 left) 2024-04-24T01:57:07.105 INFO:tasks.rados.rados.0.smithi079.stdout:434: done (0 left) 2024-04-24T01:57:07.105 INFO:tasks.rados.rados.0.smithi079.stdout:435: snap_create 2024-04-24T01:57:08.107 INFO:tasks.rados.rados.0.smithi079.stdout:435: done (0 left) 2024-04-24T01:57:08.107 INFO:tasks.rados.rados.0.smithi079.stdout:436: write oid 45 current snap is 52 2024-04-24T01:57:08.107 INFO:tasks.rados.rados.0.smithi079.stdout:436: seq_num 224 ranges {564768=630013,1674790=447398,2768515=626225} 2024-04-24T01:57:08.113 INFO:tasks.rados.rados.0.smithi079.stdout:436: writing smithi07929762-45 from 564768 to 1194781 tid 1 2024-04-24T01:57:08.119 INFO:tasks.rados.rados.0.smithi079.stdout:436: writing smithi07929762-45 from 1674790 to 2122188 tid 2 2024-04-24T01:57:08.126 INFO:tasks.rados.rados.0.smithi079.stdout:436: writing smithi07929762-45 from 2768515 to 3394740 tid 3 2024-04-24T01:57:08.126 INFO:tasks.rados.rados.0.smithi079.stdout:437: snap_create 2024-04-24T01:57:08.207 INFO:tasks.rados.rados.0.smithi079.stdout:436: finishing write tid 1 to smithi07929762-45 2024-04-24T01:57:08.217 INFO:tasks.rados.rados.0.smithi079.stdout:436: finishing write tid 2 to smithi07929762-45 2024-04-24T01:57:08.229 INFO:tasks.rados.rados.0.smithi079.stdout:436: finishing write tid 3 to smithi07929762-45 2024-04-24T01:57:08.234 INFO:tasks.rados.rados.0.smithi079.stdout:436: finishing write tid 4 to smithi07929762-45 2024-04-24T01:57:08.237 INFO:tasks.rados.rados.0.smithi079.stdout:436: finishing write tid 5 to smithi07929762-45 2024-04-24T01:57:08.237 INFO:tasks.rados.rados.0.smithi079.stdout:436: oid 45 updating version 0 to 100 2024-04-24T01:57:08.237 INFO:tasks.rados.rados.0.smithi079.stdout:436: oid 45 version 100 is already newer than 100 2024-04-24T01:57:08.237 INFO:tasks.rados.rados.0.smithi079.stdout:436: oid 45 version 100 is already newer than 100 2024-04-24T01:57:08.237 INFO:tasks.rados.rados.0.smithi079.stdout:436: oid 45 version 100 is already newer than 100 2024-04-24T01:57:08.237 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 45 v 100 (ObjNum 224 snap 52 seq_num 224) dirty exists 2024-04-24T01:57:08.237 INFO:tasks.rados.rados.0.smithi079.stdout:436: left oid 45 (ObjNum 224 snap 52 seq_num 224) 2024-04-24T01:57:08.600 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T01:57:08.706 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:08.703+0000 7f3112e0f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:08.709 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:08.709+0000 7f3112e0f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:09.121 INFO:tasks.rados.rados.0.smithi079.stdout:436: done (1 left) 2024-04-24T01:57:09.122 INFO:tasks.rados.rados.0.smithi079.stdout:437: done (0 left) 2024-04-24T01:57:09.122 INFO:tasks.rados.rados.0.smithi079.stdout:438: snap_create 2024-04-24T01:57:09.511 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-04-24T01:57:09.511 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-04-24T01:57:09.512 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T01:57:09.626 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:09.621+0000 7f80aeff5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:09.630 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:09.629+0000 7f80aeff5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:09.815 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:57:09.815 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-24T01:57:09.827 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":164,"stamp":"2024-04-24T01:57:09.117260+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":119214075,"num_objects":71,"num_object_clones":27,"num_object_copies":213,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":958,"num_read_kb":554759,"num_write":938,"num_write_kb":224070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1194,"ondisk_log_size":1194,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":426528,"kb_used_data":302592,"kb_used_omap":0,"kb_used_meta":123904,"kb_avail":377060832,"statfs":{"total":386547056640,"available":386110291968,"internally_reserved":0,"allocated":309854208,"data_stored":334812433,"data_compressed":89889,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":126877696},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":84035,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":1731,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.018639"},"pg_stats":[{"pgid":"3.f","version":"108'57","reported_seq":356,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-04-24T01:57:02.285421+0000","last_change":"2024-04-24T01:57:02.298701+0000","last_active":"2024-04-24T01:57:02.285421+0000","last_peered":"2024-04-24T01:57:02.285421+0000","last_clean":"2024-04-24T01:57:02.285421+0000","last_became_active":"2024-04-24T01:55:30.386726+0000","last_became_peered":"2024-04-24T01:55:30.386726+0000","last_unstale":"2024-04-24T01:57:02.285421+0000","last_undegraded":"2024-04-24T01:57:02.285421+0000","last_fullsized":"2024-04-24T01:57:02.285421+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0,"stat_sum":{"num_bytes":7159782,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":48,"num_read_kb":36883,"num_write":37,"num_write_kb":11452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"92'95","reported_seq":397,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-24T01:57:04.114744+0000","last_change":"2024-04-24T01:57:02.249913+0000","last_active":"2024-04-24T01:57:04.114744+0000","last_peered":"2024-04-24T01:57:04.114744+0000","last_clean":"2024-04-24T01:57:04.114744+0000","last_became_active":"2024-04-24T01:55:30.321866+0000","last_became_peered":"2024-04-24T01:55:30.321866+0000","last_unstale":"2024-04-24T01:57:04.114744+0000","last_undegraded":"2024-04-24T01:57:04.114744+0000","last_fullsized":"2024-04-24T01:57:04.114744+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":7,"snaptrim_duration":0,"stat_sum":{"num_bytes":10437114,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":77,"num_read_kb":47294,"num_write":66,"num_write_kb":17834,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"112'111","reported_seq":399,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-24T01:57:08.121884+0000","last_change":"2024-04-24T01:57:06.213244+0000","last_active":"2024-04-24T01:57:08.121884+0000","last_peered":"2024-04-24T01:57:08.121884+0000","last_clean":"2024-04-24T01:57:08.121884+0000","last_became_active":"2024-04-24T01:55:30.320186+0000","last_became_peered":"2024-04-24T01:55:30.320186+0000","last_unstale":"2024-04-24T01:57:08.121884+0000","last_undegraded":"2024-04-24T01:57:08.121884+0000","last_fullsized":"2024-04-24T01:57:08.121884+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":9,"snaptrim_duration":0,"stat_sum":{"num_bytes":7294238,"num_objects":7,"num_object_clones":4,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":48,"num_read_kb":16607,"num_write":78,"num_write_kb":21515,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"106'77","reported_seq":379,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-04-24T01:57:02.268863+0000","last_change":"2024-04-24T01:57:02.282482+0000","last_active":"2024-04-24T01:57:02.268863+0000","last_peered":"2024-04-24T01:57:02.268863+0000","last_clean":"2024-04-24T01:57:02.268863+0000","last_became_active":"2024-04-24T01:55:30.288081+0000","last_became_peered":"2024-04-24T01:55:30.288081+0000","last_unstale":"2024-04-24T01:57:02.268863+0000","last_undegraded":"2024-04-24T01:57:02.268863+0000","last_fullsized":"2024-04-24T01:57:02.268863+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":9,"snaptrim_duration":0,"stat_sum":{"num_bytes":8134588,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":73,"num_read_kb":54173,"num_write":46,"num_write_kb":13369,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"94'35","reported_seq":333,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T01:57:03.112461+0000","last_change":"2024-04-24T01:57:02.114644+0000","last_active":"2024-04-24T01:57:03.112461+0000","last_peered":"2024-04-24T01:57:03.112461+0000","last_clean":"2024-04-24T01:57:03.112461+0000","last_became_active":"2024-04-24T01:55:30.333815+0000","last_became_peered":"2024-04-24T01:55:30.333815+0000","last_unstale":"2024-04-24T01:57:03.112461+0000","last_undegraded":"2024-04-24T01:57:03.112461+0000","last_fullsized":"2024-04-24T01:57:03.112461+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0,"stat_sum":{"num_bytes":3226944,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":30,"num_read_kb":17555,"num_write":31,"num_write_kb":8358,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"106'129","reported_seq":441,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-04-24T01:57:02.271265+0000","last_change":"2024-04-24T01:57:02.280914+0000","last_active":"2024-04-24T01:57:02.271265+0000","last_peered":"2024-04-24T01:57:02.271265+0000","last_clean":"2024-04-24T01:57:02.271265+0000","last_became_active":"2024-04-24T01:55:30.315075+0000","last_became_peered":"2024-04-24T01:55:30.315075+0000","last_unstale":"2024-04-24T01:57:02.271265+0000","last_undegraded":"2024-04-24T01:57:02.271265+0000","last_fullsized":"2024-04-24T01:57:02.271265+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":15,"snaptrim_duration":0,"stat_sum":{"num_bytes":16823807,"num_objects":8,"num_object_clones":2,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":102,"num_read_kb":58594,"num_write":103,"num_write_kb":28282,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"106'35","reported_seq":327,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-04-24T01:57:02.278180+0000","last_change":"2024-04-24T01:57:02.283349+0000","last_active":"2024-04-24T01:57:02.278180+0000","last_peered":"2024-04-24T01:57:02.278180+0000","last_clean":"2024-04-24T01:57:02.278180+0000","last_became_active":"2024-04-24T01:55:30.350300+0000","last_became_peered":"2024-04-24T01:55:30.350300+0000","last_unstale":"2024-04-24T01:57:02.278180+0000","last_undegraded":"2024-04-24T01:57:02.278180+0000","last_fullsized":"2024-04-24T01:57:02.278180+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0,"stat_sum":{"num_bytes":3462242,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":17,"num_read_kb":5006,"num_write":31,"num_write_kb":7547,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"97'7","reported_seq":298,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-24T01:57:08.128018+0000","last_change":"2024-04-24T01:57:06.185249+0000","last_active":"2024-04-24T01:57:08.128018+0000","last_peered":"2024-04-24T01:57:08.128018+0000","last_clean":"2024-04-24T01:57:08.128018+0000","last_became_active":"2024-04-24T01:55:30.376813+0000","last_became_peered":"2024-04-24T01:55:30.376813+0000","last_unstale":"2024-04-24T01:57:08.128018+0000","last_undegraded":"2024-04-24T01:57:08.128018+0000","last_fullsized":"2024-04-24T01:57:08.128018+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'97","reported_seq":389,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T01:57:03.110464+0000","last_change":"2024-04-24T01:55:16.196470+0000","last_active":"2024-04-24T01:57:03.110464+0000","last_peered":"2024-04-24T01:57:03.110464+0000","last_clean":"2024-04-24T01:57:03.110464+0000","last_became_active":"2024-04-24T01:55:16.190464+0000","last_became_peered":"2024-04-24T01:55:16.190464+0000","last_unstale":"2024-04-24T01:57:03.110464+0000","last_undegraded":"2024-04-24T01:57:03.110464+0000","last_fullsized":"2024-04-24T01:57:03.110464+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:15.127115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:15.127115+0000","last_clean_scrub_stamp":"2024-04-24T01:55:15.127115+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":259,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-24T01:57:08.127474+0000","last_change":"2024-04-24T01:55:18.236148+0000","last_active":"2024-04-24T01:57:08.127474+0000","last_peered":"2024-04-24T01:57:08.127474+0000","last_clean":"2024-04-24T01:57:08.127474+0000","last_became_active":"2024-04-24T01:55:18.230154+0000","last_became_peered":"2024-04-24T01:55:18.230154+0000","last_unstale":"2024-04-24T01:57:08.127474+0000","last_undegraded":"2024-04-24T01:57:08.127474+0000","last_fullsized":"2024-04-24T01:57:08.127474+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"112'110","reported_seq":433,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-24T01:57:08.129093+0000","last_change":"2024-04-24T01:57:06.226944+0000","last_active":"2024-04-24T01:57:08.129093+0000","last_peered":"2024-04-24T01:57:08.129093+0000","last_clean":"2024-04-24T01:57:08.129093+0000","last_became_active":"2024-04-24T01:55:30.316222+0000","last_became_peered":"2024-04-24T01:55:30.316222+0000","last_unstale":"2024-04-24T01:57:08.129093+0000","last_undegraded":"2024-04-24T01:57:08.129093+0000","last_fullsized":"2024-04-24T01:57:08.129093+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":9,"snaptrim_duration":0,"stat_sum":{"num_bytes":12218689,"num_objects":7,"num_object_clones":3,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":106,"num_read_kb":71508,"num_write":84,"num_write_kb":24395,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":257,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-04-24T01:57:02.279849+0000","last_change":"2024-04-24T01:55:18.265905+0000","last_active":"2024-04-24T01:57:02.279849+0000","last_peered":"2024-04-24T01:57:02.279849+0000","last_clean":"2024-04-24T01:57:02.279849+0000","last_became_active":"2024-04-24T01:55:18.257020+0000","last_became_peered":"2024-04-24T01:55:18.257020+0000","last_unstale":"2024-04-24T01:57:02.279849+0000","last_undegraded":"2024-04-24T01:57:02.279849+0000","last_fullsized":"2024-04-24T01:57:02.279849+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"106'72","reported_seq":374,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T01:57:03.105426+0000","last_change":"2024-04-24T01:57:02.108260+0000","last_active":"2024-04-24T01:57:03.105426+0000","last_peered":"2024-04-24T01:57:03.105426+0000","last_clean":"2024-04-24T01:57:03.105426+0000","last_became_active":"2024-04-24T01:55:30.344129+0000","last_became_peered":"2024-04-24T01:55:30.344129+0000","last_unstale":"2024-04-24T01:57:03.105426+0000","last_undegraded":"2024-04-24T01:57:03.105426+0000","last_fullsized":"2024-04-24T01:57:03.105426+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0,"stat_sum":{"num_bytes":9714446,"num_objects":7,"num_object_clones":4,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":63,"num_read_kb":40691,"num_write":50,"num_write_kb":14326,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":259,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-24T01:57:08.116463+0000","last_change":"2024-04-24T01:55:18.215251+0000","last_active":"2024-04-24T01:57:08.116463+0000","last_peered":"2024-04-24T01:57:08.116463+0000","last_clean":"2024-04-24T01:57:08.116463+0000","last_became_active":"2024-04-24T01:55:18.206605+0000","last_became_peered":"2024-04-24T01:55:18.206605+0000","last_unstale":"2024-04-24T01:57:08.116463+0000","last_undegraded":"2024-04-24T01:57:08.116463+0000","last_fullsized":"2024-04-24T01:57:08.116463+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"95'36","reported_seq":332,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T01:57:03.114708+0000","last_change":"2024-04-24T01:57:02.110567+0000","last_active":"2024-04-24T01:57:03.114708+0000","last_peered":"2024-04-24T01:57:03.114708+0000","last_clean":"2024-04-24T01:57:03.114708+0000","last_became_active":"2024-04-24T01:55:30.401366+0000","last_became_peered":"2024-04-24T01:55:30.401366+0000","last_unstale":"2024-04-24T01:57:03.114708+0000","last_undegraded":"2024-04-24T01:57:03.114708+0000","last_fullsized":"2024-04-24T01:57:03.114708+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0,"stat_sum":{"num_bytes":7176556,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":32,"num_read_kb":25812,"num_write":24,"num_write_kb":6694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":257,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-04-24T01:57:02.281099+0000","last_change":"2024-04-24T01:55:18.230893+0000","last_active":"2024-04-24T01:57:02.281099+0000","last_peered":"2024-04-24T01:57:02.281099+0000","last_clean":"2024-04-24T01:57:02.281099+0000","last_became_active":"2024-04-24T01:55:18.225188+0000","last_became_peered":"2024-04-24T01:55:18.225188+0000","last_unstale":"2024-04-24T01:57:02.281099+0000","last_undegraded":"2024-04-24T01:57:02.281099+0000","last_fullsized":"2024-04-24T01:57:02.281099+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"83'64","reported_seq":361,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-24T01:57:04.110742+0000","last_change":"2024-04-24T01:57:02.250230+0000","last_active":"2024-04-24T01:57:04.110742+0000","last_peered":"2024-04-24T01:57:04.110742+0000","last_clean":"2024-04-24T01:57:04.110742+0000","last_became_active":"2024-04-24T01:55:30.343318+0000","last_became_peered":"2024-04-24T01:55:30.343318+0000","last_unstale":"2024-04-24T01:57:04.110742+0000","last_undegraded":"2024-04-24T01:57:04.110742+0000","last_fullsized":"2024-04-24T01:57:04.110742+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":7,"snaptrim_duration":0,"stat_sum":{"num_bytes":6571504,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46778,"num_write":46,"num_write_kb":13987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"96'91","reported_seq":400,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-24T01:57:08.127283+0000","last_change":"2024-04-24T01:57:06.187403+0000","last_active":"2024-04-24T01:57:08.127283+0000","last_peered":"2024-04-24T01:57:08.127283+0000","last_clean":"2024-04-24T01:57:08.127283+0000","last_became_active":"2024-04-24T01:55:30.345308+0000","last_became_peered":"2024-04-24T01:55:30.345308+0000","last_unstale":"2024-04-24T01:57:08.127283+0000","last_undegraded":"2024-04-24T01:57:08.127283+0000","last_fullsized":"2024-04-24T01:57:08.127283+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":8,"snaptrim_duration":0,"stat_sum":{"num_bytes":10773287,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":79,"num_read_kb":46934,"num_write":64,"num_write_kb":16455,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":253,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-24T01:57:04.118915+0000","last_change":"2024-04-24T01:55:18.224238+0000","last_active":"2024-04-24T01:57:04.118915+0000","last_peered":"2024-04-24T01:57:04.118915+0000","last_clean":"2024-04-24T01:57:04.118915+0000","last_became_active":"2024-04-24T01:55:18.219131+0000","last_became_peered":"2024-04-24T01:55:18.219131+0000","last_unstale":"2024-04-24T01:57:04.118915+0000","last_undegraded":"2024-04-24T01:57:04.118915+0000","last_fullsized":"2024-04-24T01:57:04.118915+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"92'43","reported_seq":329,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-24T01:57:04.122199+0000","last_change":"2024-04-24T01:57:02.259408+0000","last_active":"2024-04-24T01:57:04.122199+0000","last_peered":"2024-04-24T01:57:04.122199+0000","last_clean":"2024-04-24T01:57:04.122199+0000","last_became_active":"2024-04-24T01:55:30.319958+0000","last_became_peered":"2024-04-24T01:55:30.319958+0000","last_unstale":"2024-04-24T01:57:04.122199+0000","last_undegraded":"2024-04-24T01:57:04.122199+0000","last_fullsized":"2024-04-24T01:57:04.122199+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":22,"num_read_kb":19606,"num_write":28,"num_write_kb":7720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":256,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T01:57:03.115774+0000","last_change":"2024-04-24T01:55:18.235763+0000","last_active":"2024-04-24T01:57:03.115774+0000","last_peered":"2024-04-24T01:57:03.115774+0000","last_clean":"2024-04-24T01:57:03.115774+0000","last_became_active":"2024-04-24T01:55:18.229175+0000","last_became_peered":"2024-04-24T01:55:18.229175+0000","last_unstale":"2024-04-24T01:57:03.115774+0000","last_undegraded":"2024-04-24T01:57:03.115774+0000","last_fullsized":"2024-04-24T01:57:03.115774+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"68'28","reported_seq":328,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T01:57:03.096498+0000","last_change":"2024-04-24T01:57:02.108507+0000","last_active":"2024-04-24T01:57:03.096498+0000","last_peered":"2024-04-24T01:57:03.096498+0000","last_clean":"2024-04-24T01:57:03.096498+0000","last_became_active":"2024-04-24T01:55:30.372690+0000","last_became_peered":"2024-04-24T01:55:30.372690+0000","last_unstale":"2024-04-24T01:57:03.096498+0000","last_undegraded":"2024-04-24T01:57:03.096498+0000","last_fullsized":"2024-04-24T01:57:03.096498+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":3,"snaptrim_duration":0,"stat_sum":{"num_bytes":2521050,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":34,"num_read_kb":24087,"num_write":24,"num_write_kb":6609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":253,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-24T01:57:04.123334+0000","last_change":"2024-04-24T01:55:18.278351+0000","last_active":"2024-04-24T01:57:04.123334+0000","last_peered":"2024-04-24T01:57:04.123334+0000","last_clean":"2024-04-24T01:57:04.123334+0000","last_became_active":"2024-04-24T01:55:18.273668+0000","last_became_peered":"2024-04-24T01:55:18.273668+0000","last_unstale":"2024-04-24T01:57:04.123334+0000","last_undegraded":"2024-04-24T01:57:04.123334+0000","last_fullsized":"2024-04-24T01:57:04.123334+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"108'107","reported_seq":404,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-24T01:57:04.121546+0000","last_change":"2024-04-24T01:57:02.277453+0000","last_active":"2024-04-24T01:57:04.121546+0000","last_peered":"2024-04-24T01:57:04.121546+0000","last_clean":"2024-04-24T01:57:04.121546+0000","last_became_active":"2024-04-24T01:55:30.363123+0000","last_became_peered":"2024-04-24T01:55:30.363123+0000","last_unstale":"2024-04-24T01:57:04.121546+0000","last_undegraded":"2024-04-24T01:57:04.121546+0000","last_fullsized":"2024-04-24T01:57:04.121546+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":7,"snaptrim_duration":0,"stat_sum":{"num_bytes":13109460,"num_objects":9,"num_object_clones":5,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":69,"num_read_kb":43139,"num_write":84,"num_write_kb":22785,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":259,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-24T01:57:08.126041+0000","last_change":"2024-04-24T01:55:18.241117+0000","last_active":"2024-04-24T01:57:08.126041+0000","last_peered":"2024-04-24T01:57:08.126041+0000","last_clean":"2024-04-24T01:57:08.126041+0000","last_became_active":"2024-04-24T01:55:18.234313+0000","last_became_peered":"2024-04-24T01:55:18.234313+0000","last_unstale":"2024-04-24T01:57:08.126041+0000","last_undegraded":"2024-04-24T01:57:08.126041+0000","last_fullsized":"2024-04-24T01:57:08.126041+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":118623707,"num_objects":69,"num_object_clones":27,"num_object_copies":207,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":855,"num_read_kb":554668,"num_write":801,"num_write_kb":222176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":307302400,"data_stored":336746561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1097,"ondisk_log_size":1097,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":1}],"osd_stats":[{"osd":3,"up_from":10,"seq":42949673069,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":115864,"kb_used_data":84880,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94255976,"statfs":{"total":96636764160,"available":96518119424,"internally_reserved":0,"allocated":86917120,"data_stored":94714938,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673069,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":90128,"kb_used_data":59144,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94281712,"statfs":{"total":96636764160,"available":96544473088,"internally_reserved":0,"allocated":60563456,"data_stored":63146809,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705774,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":98200,"kb_used_data":67216,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94273640,"statfs":{"total":96636764160,"available":96536207360,"internally_reserved":0,"allocated":68829184,"data_stored":74017177,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705778,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":122336,"kb_used_data":91352,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94249504,"statfs":{"total":96636764160,"available":96511492096,"internally_reserved":0,"allocated":93544448,"data_stored":102933509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":67878912,"data_stored":73055121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":92614656,"data_stored":102543500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":60231680,"data_stored":64791732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":86577152,"data_stored":96356208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T01:57:10.119 INFO:tasks.rados.rados.0.smithi079.stdout:438: done (0 left) 2024-04-24T01:57:10.119 INFO:tasks.rados.rados.0.smithi079.stdout:439: delete oid 13 current snap is 54 2024-04-24T01:57:10.178 INFO:tasks.rados.rados.0.smithi079.stdout:439: done (0 left) 2024-04-24T01:57:10.178 INFO:tasks.rados.rados.0.smithi079.stdout:440: write (excl) oid 36 current snap is 54 2024-04-24T01:57:10.178 INFO:tasks.rados.rados.0.smithi079.stdout:440: seq_num 226 ranges {598382=780511,2075401=451067,3107008=648769} 2024-04-24T01:57:10.185 INFO:tasks.rados.rados.0.smithi079.stdout:440: writing smithi07929762-36 from 598382 to 1378893 tid 1 2024-04-24T01:57:10.191 INFO:tasks.rados.rados.0.smithi079.stdout:440: writing smithi07929762-36 from 2075401 to 2526468 tid 2 2024-04-24T01:57:10.198 INFO:tasks.rados.rados.0.smithi079.stdout:440: writing smithi07929762-36 from 3107008 to 3755777 tid 3 2024-04-24T01:57:10.198 INFO:tasks.rados.rados.0.smithi079.stdout:441: snap_create 2024-04-24T01:57:10.291 INFO:tasks.rados.rados.0.smithi079.stdout:440: finishing write tid 1 to smithi07929762-36 2024-04-24T01:57:10.308 INFO:tasks.rados.rados.0.smithi079.stdout:440: finishing write tid 2 to smithi07929762-36 2024-04-24T01:57:10.313 INFO:tasks.rados.rados.0.smithi079.stdout:440: finishing write tid 3 to smithi07929762-36 2024-04-24T01:57:10.318 INFO:tasks.rados.rados.0.smithi079.stdout:440: finishing write tid 4 to smithi07929762-36 2024-04-24T01:57:10.320 INFO:tasks.rados.rados.0.smithi079.stdout:440: finishing write tid 5 to smithi07929762-36 2024-04-24T01:57:10.320 INFO:tasks.rados.rados.0.smithi079.stdout:440: oid 36 updating version 0 to 37 2024-04-24T01:57:10.321 INFO:tasks.rados.rados.0.smithi079.stdout:440: oid 36 updating version 37 to 40 2024-04-24T01:57:10.321 INFO:tasks.rados.rados.0.smithi079.stdout:440: oid 36 version 40 is already newer than 40 2024-04-24T01:57:10.321 INFO:tasks.rados.rados.0.smithi079.stdout:440: oid 36 version 40 is already newer than 40 2024-04-24T01:57:10.321 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 36 v 40 (ObjNum 226 snap 54 seq_num 226) dirty exists 2024-04-24T01:57:10.321 INFO:tasks.rados.rados.0.smithi079.stdout:440: left oid 36 (ObjNum 226 snap 54 seq_num 226) 2024-04-24T01:57:10.803 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T01:57:10.908 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:10.905+0000 7f6964aed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:10.912 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:10.911+0000 7f6964aed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:11.131 INFO:tasks.rados.rados.0.smithi079.stdout:440: done (1 left) 2024-04-24T01:57:11.131 INFO:tasks.rados.rados.0.smithi079.stdout:441: done (0 left) 2024-04-24T01:57:11.131 INFO:tasks.rados.rados.0.smithi079.stdout:442: read oid 48 snap -1 2024-04-24T01:57:11.131 INFO:tasks.rados.rados.0.smithi079.stdout:442: expect (ObjNum 217 snap 47 seq_num 217) 2024-04-24T01:57:11.131 INFO:tasks.rados.rados.0.smithi079.stdout:443: read oid 43 snap 42 2024-04-24T01:57:11.131 INFO:tasks.rados.rados.0.smithi079.stdout:443: expect deleted 2024-04-24T01:57:11.131 INFO:tasks.rados.rados.0.smithi079.stdout:444: write oid 50 current snap is 55 2024-04-24T01:57:11.131 INFO:tasks.rados.rados.0.smithi079.stdout:444: seq_num 227 ranges {615189=455760,1733883=711078,3134431=460041} 2024-04-24T01:57:11.136 INFO:tasks.rados.rados.0.smithi079.stdout:444: writing smithi07929762-50 from 615189 to 1070949 tid 1 2024-04-24T01:57:11.144 INFO:tasks.rados.rados.0.smithi079.stdout:444: writing smithi07929762-50 from 1733883 to 2444961 tid 2 2024-04-24T01:57:11.153 INFO:tasks.rados.rados.0.smithi079.stdout:444: writing smithi07929762-50 from 3134431 to 3594472 tid 3 2024-04-24T01:57:11.153 INFO:tasks.rados.rados.0.smithi079.stdout:445: read oid 1 snap -1 2024-04-24T01:57:11.153 INFO:tasks.rados.rados.0.smithi079.stdout:445: expect (ObjNum 133 snap 15 seq_num 133) 2024-04-24T01:57:11.153 INFO:tasks.rados.rados.0.smithi079.stdout:446: write oid 29 current snap is 55 2024-04-24T01:57:11.153 INFO:tasks.rados.rados.0.smithi079.stdout:446: seq_num 228 ranges {631996=531009,1792365=454736,2761854=671313} 2024-04-24T01:57:11.159 INFO:tasks.rados.rados.0.smithi079.stdout:446: writing smithi07929762-29 from 631996 to 1163005 tid 1 2024-04-24T01:57:11.169 INFO:tasks.rados.rados.0.smithi079.stdout:446: writing smithi07929762-29 from 1792365 to 2247101 tid 2 2024-04-24T01:57:11.176 INFO:tasks.rados.rados.0.smithi079.stdout:446: writing smithi07929762-29 from 2761854 to 3433167 tid 3 2024-04-24T01:57:11.176 INFO:tasks.rados.rados.0.smithi079.stdout:447: snap_remove snap 49 2024-04-24T01:57:12.136 INFO:tasks.rados.rados.0.smithi079.stdout:447: done (5 left) 2024-04-24T01:57:12.137 INFO:tasks.rados.rados.0.smithi079.stdout:448: delete oid 12 current snap is 55 2024-04-24T01:57:12.137 INFO:tasks.rados.rados.0.smithi079.stdout:446: finishing write tid 1 to smithi07929762-29 2024-04-24T01:57:12.137 INFO:tasks.rados.rados.0.smithi079.stdout:446: finishing write tid 2 to smithi07929762-29 2024-04-24T01:57:12.137 INFO:tasks.rados.rados.0.smithi079.stdout:446: finishing write tid 3 to smithi07929762-29 2024-04-24T01:57:12.137 INFO:tasks.rados.rados.0.smithi079.stdout:446: finishing write tid 4 to smithi07929762-29 2024-04-24T01:57:12.137 INFO:tasks.rados.rados.0.smithi079.stdout:446: finishing write tid 5 to smithi07929762-29 2024-04-24T01:57:12.137 INFO:tasks.rados.rados.0.smithi079.stdout:446: oid 29 updating version 0 to 115 2024-04-24T01:57:12.137 INFO:tasks.rados.rados.0.smithi079.stdout:446: oid 29 version 115 is already newer than 115 2024-04-24T01:57:12.137 INFO:tasks.rados.rados.0.smithi079.stdout:446: oid 29 version 115 is already newer than 115 2024-04-24T01:57:12.137 INFO:tasks.rados.rados.0.smithi079.stdout:446: oid 29 version 115 is already newer than 115 2024-04-24T01:57:12.137 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 29 v 115 (ObjNum 228 snap 55 seq_num 228) dirty exists 2024-04-24T01:57:12.137 INFO:tasks.rados.rados.0.smithi079.stdout:446: left oid 29 (ObjNum 228 snap 55 seq_num 228) 2024-04-24T01:57:12.137 INFO:tasks.rados.rados.0.smithi079.stdout:444: finishing write tid 1 to smithi07929762-50 2024-04-24T01:57:12.137 INFO:tasks.rados.rados.0.smithi079.stdout:442: expect (ObjNum 217 snap 47 seq_num 217) 2024-04-24T01:57:12.369 INFO:tasks.rados.rados.0.smithi079.stdout:444: finishing write tid 2 to smithi07929762-50 2024-04-24T01:57:12.369 INFO:tasks.rados.rados.0.smithi079.stdout:444: finishing write tid 3 to smithi07929762-50 2024-04-24T01:57:12.369 INFO:tasks.rados.rados.0.smithi079.stdout:444: finishing write tid 4 to smithi07929762-50 2024-04-24T01:57:12.369 INFO:tasks.rados.rados.0.smithi079.stdout:445: expect (ObjNum 133 snap 15 seq_num 133) 2024-04-24T01:57:12.533 INFO:tasks.rados.rados.0.smithi079.stdout:444: finishing write tid 5 to smithi07929762-50 2024-04-24T01:57:12.533 INFO:tasks.rados.rados.0.smithi079.stdout:444: oid 50 updating version 0 to 82 2024-04-24T01:57:12.533 INFO:tasks.rados.rados.0.smithi079.stdout:444: oid 50 version 82 is already newer than 82 2024-04-24T01:57:12.533 INFO:tasks.rados.rados.0.smithi079.stdout:444: oid 50 version 82 is already newer than 82 2024-04-24T01:57:12.533 INFO:tasks.rados.rados.0.smithi079.stdout:444: oid 50 version 82 is already newer than 82 2024-04-24T01:57:12.533 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 50 v 82 (ObjNum 227 snap 55 seq_num 227) dirty exists 2024-04-24T01:57:12.533 INFO:tasks.rados.rados.0.smithi079.stdout:444: left oid 50 (ObjNum 227 snap 55 seq_num 227) 2024-04-24T01:57:12.534 INFO:tasks.rados.rados.0.smithi079.stdout:442: done (5 left) 2024-04-24T01:57:12.534 INFO:tasks.rados.rados.0.smithi079.stdout:443: done (4 left) 2024-04-24T01:57:12.534 INFO:tasks.rados.rados.0.smithi079.stdout:444: done (3 left) 2024-04-24T01:57:12.534 INFO:tasks.rados.rados.0.smithi079.stdout:445: done (2 left) 2024-04-24T01:57:12.534 INFO:tasks.rados.rados.0.smithi079.stdout:446: done (1 left) 2024-04-24T01:57:12.534 INFO:tasks.rados.rados.0.smithi079.stdout:448: done (0 left) 2024-04-24T01:57:12.534 INFO:tasks.rados.rados.0.smithi079.stdout:449: read oid 29 snap 47 2024-04-24T01:57:12.534 INFO:tasks.rados.rados.0.smithi079.stdout:449: expect deleted 2024-04-24T01:57:12.534 INFO:tasks.rados.rados.0.smithi079.stdout:450: read oid 37 snap -1 2024-04-24T01:57:12.534 INFO:tasks.rados.rados.0.smithi079.stdout:450: expect deleted 2024-04-24T01:57:12.534 INFO:tasks.rados.rados.0.smithi079.stdout:451: write (excl) oid 18 current snap is 55 2024-04-24T01:57:12.534 INFO:tasks.rados.rados.0.smithi079.stdout:451: seq_num 230 ranges {665610=797860,1909329=458405,2816700=410210} 2024-04-24T01:57:12.542 INFO:tasks.rados.rados.0.smithi079.stdout:451: writing smithi07929762-18 from 665610 to 1463470 tid 1 2024-04-24T01:57:12.548 INFO:tasks.rados.rados.0.smithi079.stdout:451: writing smithi07929762-18 from 1909329 to 2367734 tid 2 2024-04-24T01:57:12.554 INFO:tasks.rados.rados.0.smithi079.stdout:451: writing smithi07929762-18 from 2816700 to 3226910 tid 3 2024-04-24T01:57:12.554 INFO:tasks.rados.rados.0.smithi079.stdout:452: read oid 44 snap 47 2024-04-24T01:57:12.554 INFO:tasks.rados.rados.0.smithi079.stdout:452: expect (ObjNum 105 snap 9 seq_num 105) 2024-04-24T01:57:12.554 INFO:tasks.rados.rados.0.smithi079.stdout:453: write (excl) oid 39 current snap is 55 2024-04-24T01:57:12.554 INFO:tasks.rados.rados.0.smithi079.stdout:453: seq_num 231 ranges {682417=473109,1567811=602063,2844123=621482} 2024-04-24T01:57:12.562 INFO:tasks.rados.rados.0.smithi079.stdout:453: writing smithi07929762-39 from 682417 to 1155526 tid 1 2024-04-24T01:57:12.571 INFO:tasks.rados.rados.0.smithi079.stdout:453: writing smithi07929762-39 from 1567811 to 2169874 tid 2 2024-04-24T01:57:12.582 INFO:tasks.rados.rados.0.smithi079.stdout:453: writing smithi07929762-39 from 2844123 to 3465605 tid 3 2024-04-24T01:57:12.582 INFO:tasks.rados.rados.0.smithi079.stdout:454: write oid 13 current snap is 55 2024-04-24T01:57:12.582 INFO:tasks.rados.rados.0.smithi079.stdout:454: seq_num 232 ranges {699224=548358,2026293=462074,3271546=432754} 2024-04-24T01:57:12.590 INFO:tasks.rados.rados.0.smithi079.stdout:454: writing smithi07929762-13 from 699224 to 1247582 tid 1 2024-04-24T01:57:12.599 INFO:tasks.rados.rados.0.smithi079.stdout:454: writing smithi07929762-13 from 2026293 to 2488367 tid 2 2024-04-24T01:57:12.607 INFO:tasks.rados.rados.0.smithi079.stdout:454: writing smithi07929762-13 from 3271546 to 3704300 tid 3 2024-04-24T01:57:12.608 INFO:tasks.rados.rados.0.smithi079.stdout:455: snap_create 2024-04-24T01:57:12.608 INFO:tasks.rados.rados.0.smithi079.stdout:451: finishing write tid 1 to smithi07929762-18 2024-04-24T01:57:12.610 INFO:tasks.rados.rados.0.smithi079.stdout:452: expect (ObjNum 105 snap 9 seq_num 105) 2024-04-24T01:57:12.850 INFO:tasks.rados.rados.0.smithi079.stdout:453: finishing write tid 1 to smithi07929762-39 2024-04-24T01:57:12.850 INFO:tasks.rados.rados.0.smithi079.stdout:451: finishing write tid 2 to smithi07929762-18 2024-04-24T01:57:12.850 INFO:tasks.rados.rados.0.smithi079.stdout:451: finishing write tid 3 to smithi07929762-18 2024-04-24T01:57:12.850 INFO:tasks.rados.rados.0.smithi079.stdout:451: finishing write tid 4 to smithi07929762-18 2024-04-24T01:57:12.850 INFO:tasks.rados.rados.0.smithi079.stdout:454: finishing write tid 1 to smithi07929762-13 2024-04-24T01:57:12.850 INFO:tasks.rados.rados.0.smithi079.stdout:453: finishing write tid 2 to smithi07929762-39 2024-04-24T01:57:12.850 INFO:tasks.rados.rados.0.smithi079.stdout:451: finishing write tid 5 to smithi07929762-18 2024-04-24T01:57:12.851 INFO:tasks.rados.rados.0.smithi079.stdout:451: oid 18 updating version 0 to 133 2024-04-24T01:57:12.851 INFO:tasks.rados.rados.0.smithi079.stdout:451: oid 18 updating version 133 to 136 2024-04-24T01:57:12.851 INFO:tasks.rados.rados.0.smithi079.stdout:451: oid 18 version 136 is already newer than 136 2024-04-24T01:57:12.851 INFO:tasks.rados.rados.0.smithi079.stdout:451: oid 18 version 136 is already newer than 136 2024-04-24T01:57:12.851 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 18 v 136 (ObjNum 230 snap 55 seq_num 230) dirty exists 2024-04-24T01:57:12.851 INFO:tasks.rados.rados.0.smithi079.stdout:451: left oid 18 (ObjNum 230 snap 55 seq_num 230) 2024-04-24T01:57:12.851 INFO:tasks.rados.rados.0.smithi079.stdout:454: finishing write tid 2 to smithi07929762-13 2024-04-24T01:57:12.851 INFO:tasks.rados.rados.0.smithi079.stdout:453: finishing write tid 3 to smithi07929762-39 2024-04-24T01:57:12.851 INFO:tasks.rados.rados.0.smithi079.stdout:454: finishing write tid 3 to smithi07929762-13 2024-04-24T01:57:12.851 INFO:tasks.rados.rados.0.smithi079.stdout:454: finishing write tid 4 to smithi07929762-13 2024-04-24T01:57:12.851 INFO:tasks.rados.rados.0.smithi079.stdout:454: finishing write tid 5 to smithi07929762-13 2024-04-24T01:57:12.851 INFO:tasks.rados.rados.0.smithi079.stdout:454: oid 13 updating version 0 to 118 2024-04-24T01:57:12.851 INFO:tasks.rados.rados.0.smithi079.stdout:454: oid 13 version 118 is already newer than 118 2024-04-24T01:57:12.851 INFO:tasks.rados.rados.0.smithi079.stdout:454: oid 13 version 118 is already newer than 118 2024-04-24T01:57:12.851 INFO:tasks.rados.rados.0.smithi079.stdout:454: oid 13 version 118 is already newer than 118 2024-04-24T01:57:12.852 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 13 v 118 (ObjNum 232 snap 55 seq_num 232) dirty exists 2024-04-24T01:57:12.852 INFO:tasks.rados.rados.0.smithi079.stdout:454: left oid 13 (ObjNum 232 snap 55 seq_num 232) 2024-04-24T01:57:12.852 INFO:tasks.rados.rados.0.smithi079.stdout:453: finishing write tid 4 to smithi07929762-39 2024-04-24T01:57:12.852 INFO:tasks.rados.rados.0.smithi079.stdout:453: finishing write tid 5 to smithi07929762-39 2024-04-24T01:57:12.852 INFO:tasks.rados.rados.0.smithi079.stdout:453: oid 39 updating version 0 to 65 2024-04-24T01:57:12.852 INFO:tasks.rados.rados.0.smithi079.stdout:453: oid 39 updating version 65 to 68 2024-04-24T01:57:12.852 INFO:tasks.rados.rados.0.smithi079.stdout:453: oid 39 version 68 is already newer than 68 2024-04-24T01:57:12.852 INFO:tasks.rados.rados.0.smithi079.stdout:453: oid 39 version 68 is already newer than 68 2024-04-24T01:57:12.852 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 39 v 68 (ObjNum 231 snap 55 seq_num 231) dirty exists 2024-04-24T01:57:12.852 INFO:tasks.rados.rados.0.smithi079.stdout:453: left oid 39 (ObjNum 231 snap 55 seq_num 231) 2024-04-24T01:57:13.006 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T01:57:13.111 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:13.108+0000 7f8754fff640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:13.114 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:13.114+0000 7f874f59e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:13.146 INFO:tasks.rados.rados.0.smithi079.stdout:449: done (6 left) 2024-04-24T01:57:13.146 INFO:tasks.rados.rados.0.smithi079.stdout:450: done (5 left) 2024-04-24T01:57:13.146 INFO:tasks.rados.rados.0.smithi079.stdout:451: done (4 left) 2024-04-24T01:57:13.146 INFO:tasks.rados.rados.0.smithi079.stdout:452: done (3 left) 2024-04-24T01:57:13.146 INFO:tasks.rados.rados.0.smithi079.stdout:453: done (2 left) 2024-04-24T01:57:13.146 INFO:tasks.rados.rados.0.smithi079.stdout:454: done (1 left) 2024-04-24T01:57:13.147 INFO:tasks.rados.rados.0.smithi079.stdout:455: done (0 left) 2024-04-24T01:57:13.147 INFO:tasks.rados.rados.0.smithi079.stdout:456: read oid 33 snap 55 2024-04-24T01:57:13.147 INFO:tasks.rados.rados.0.smithi079.stdout:456: expect (ObjNum 200 snap 41 seq_num 200) 2024-04-24T01:57:13.147 INFO:tasks.rados.rados.0.smithi079.stdout:457: snap_create 2024-04-24T01:57:13.212 INFO:tasks.rados.rados.0.smithi079.stdout:456: expect (ObjNum 200 snap 41 seq_num 200) 2024-04-24T01:57:14.144 INFO:tasks.rados.rados.0.smithi079.stdout:456: done (1 left) 2024-04-24T01:57:14.144 INFO:tasks.rados.rados.0.smithi079.stdout:457: done (0 left) 2024-04-24T01:57:14.145 INFO:tasks.rados.rados.0.smithi079.stdout:458: write (excl) oid 45 current snap is 57 2024-04-24T01:57:14.145 INFO:tasks.rados.rados.0.smithi079.stdout:458: seq_num 233 ranges {716031=623607,2084775=605732,3298969=617062} 2024-04-24T01:57:14.152 INFO:tasks.rados.rados.0.smithi079.stdout:458: writing smithi07929762-45 from 716031 to 1339638 tid 1 2024-04-24T01:57:14.159 INFO:tasks.rados.rados.0.smithi079.stdout:458: writing smithi07929762-45 from 2084775 to 2690507 tid 2 2024-04-24T01:57:14.169 INFO:tasks.rados.rados.0.smithi079.stdout:458: writing smithi07929762-45 from 3298969 to 3916031 tid 3 2024-04-24T01:57:14.169 INFO:tasks.rados.rados.0.smithi079.stdout:459: snap_create 2024-04-24T01:57:14.245 INFO:tasks.rados.rados.0.smithi079.stdout:458: finishing write tid 1 to smithi07929762-45 2024-04-24T01:57:14.268 INFO:tasks.rados.rados.0.smithi079.stdout:458: finishing write tid 2 to smithi07929762-45 2024-04-24T01:57:14.278 INFO:tasks.rados.rados.0.smithi079.stdout:458: finishing write tid 3 to smithi07929762-45 2024-04-24T01:57:14.280 INFO:tasks.rados.rados.0.smithi079.stdout:458: finishing write tid 4 to smithi07929762-45 2024-04-24T01:57:14.281 INFO:tasks.rados.rados.0.smithi079.stdout:458: finishing write tid 5 to smithi07929762-45 2024-04-24T01:57:14.281 INFO:tasks.rados.rados.0.smithi079.stdout:458: oid 45 updating version 0 to 107 2024-04-24T01:57:14.282 INFO:tasks.rados.rados.0.smithi079.stdout:458: oid 45 version 107 is already newer than 107 2024-04-24T01:57:14.282 INFO:tasks.rados.rados.0.smithi079.stdout:458: oid 45 version 107 is already newer than 107 2024-04-24T01:57:14.282 INFO:tasks.rados.rados.0.smithi079.stdout:458: oid 45 version 107 is already newer than 104 2024-04-24T01:57:14.282 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 45 v 107 (ObjNum 233 snap 57 seq_num 233) dirty exists 2024-04-24T01:57:14.282 INFO:tasks.rados.rados.0.smithi079.stdout:458: left oid 45 (ObjNum 233 snap 57 seq_num 233) 2024-04-24T01:57:14.830 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-04-24T01:57:14.830 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-04-24T01:57:14.830 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-04-24T01:57:14.830 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-04-24T01:57:14.939 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:14.933+0000 7f27f6ead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:14.942 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:14.942+0000 7f27f6ead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:14.956 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T01:57:14.956 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T01:57:14.957 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T01:57:14.957 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T01:57:14.957 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T01:57:14.957 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T01:57:14.957 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T01:57:14.957 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T01:57:14.957 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T01:57:14.957 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T01:57:14.957 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T01:57:14.957 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T01:57:14.959 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T01:57:14.960 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T01:57:14.960 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-04-24T01:57:15.066 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:15.060+0000 7f05bd79d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:15.070 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:15.069+0000 7f05bd79d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:15.084 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T01:57:15.084 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T01:57:15.084 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T01:57:15.084 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T01:57:15.084 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T01:57:15.084 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T01:57:15.084 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T01:57:15.084 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T01:57:15.084 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T01:57:15.084 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T01:57:15.084 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T01:57:15.085 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T01:57:15.088 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T01:57:15.088 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T01:57:15.088 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-04-24T01:57:15.160 INFO:tasks.rados.rados.0.smithi079.stdout:458: done (1 left) 2024-04-24T01:57:15.160 INFO:tasks.rados.rados.0.smithi079.stdout:459: done (0 left) 2024-04-24T01:57:15.160 INFO:tasks.rados.rados.0.smithi079.stdout:460: read oid 35 snap 52 2024-04-24T01:57:15.160 INFO:tasks.rados.rados.0.smithi079.stdout:460: expect (ObjNum 188 snap 36 seq_num 188) 2024-04-24T01:57:15.160 INFO:tasks.rados.rados.0.smithi079.stdout:461: read oid 2 snap -1 2024-04-24T01:57:15.160 INFO:tasks.rados.rados.0.smithi079.stdout:461: expect deleted 2024-04-24T01:57:15.160 INFO:tasks.rados.rados.0.smithi079.stdout:462: snap_remove snap 48 2024-04-24T01:57:15.209 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:15.203+0000 7f0471e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:15.213 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:15.213+0000 7f0471e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:15.224 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T01:57:15.230 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T01:57:15.230 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T01:57:15.230 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T01:57:15.230 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T01:57:15.230 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T01:57:15.230 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T01:57:15.230 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T01:57:15.230 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T01:57:15.231 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T01:57:15.231 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T01:57:15.231 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T01:57:15.231 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T01:57:15.233 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T01:57:15.233 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T01:57:15.233 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-04-24T01:57:15.329 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:15.327+0000 7f517e06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:15.333 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:15.332+0000 7f517e06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:15.396 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:15.390+0000 7f721726d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:15.400 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:15.399+0000 7f721726d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:15.417 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T01:57:15.417 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T01:57:15.417 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T01:57:15.417 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T01:57:15.417 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T01:57:15.417 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T01:57:15.417 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T01:57:15.417 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T01:57:15.418 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T01:57:15.418 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T01:57:15.418 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T01:57:15.418 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T01:57:15.420 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T01:57:15.420 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T01:57:16.029 INFO:tasks.rados.rados.0.smithi079.stdout:462: done (2 left) 2024-04-24T01:57:16.029 INFO:tasks.rados.rados.0.smithi079.stdout:463: read oid 9 snap -1 2024-04-24T01:57:16.029 INFO:tasks.rados.rados.0.smithi079.stdout:463: expect (ObjNum 165 snap 25 seq_num 165) 2024-04-24T01:57:16.029 INFO:tasks.rados.rados.0.smithi079.stdout:464: read oid 40 snap -1 2024-04-24T01:57:16.029 INFO:tasks.rados.rados.0.smithi079.stdout:464: expect (ObjNum 122 snap 14 seq_num 122) 2024-04-24T01:57:16.029 INFO:tasks.rados.rados.0.smithi079.stdout:465: write (excl) oid 24 current snap is 58 2024-04-24T01:57:16.029 INFO:tasks.rados.rados.0.smithi079.stdout:465: seq_num 234 ranges {732838=698856,2026904=465743,3210039=571651} 2024-04-24T01:57:16.038 INFO:tasks.rados.rados.0.smithi079.stdout:465: writing smithi07929762-24 from 732838 to 1431694 tid 1 2024-04-24T01:57:16.046 INFO:tasks.rados.rados.0.smithi079.stdout:465: writing smithi07929762-24 from 2026904 to 2492647 tid 2 2024-04-24T01:57:16.056 INFO:tasks.rados.rados.0.smithi079.stdout:465: writing smithi07929762-24 from 3210039 to 3781690 tid 3 2024-04-24T01:57:16.056 INFO:tasks.rados.rados.0.smithi079.stdout:466: read oid 17 snap -1 2024-04-24T01:57:16.056 INFO:tasks.rados.rados.0.smithi079.stdout:466: expect (ObjNum 220 snap 49 seq_num 220) 2024-04-24T01:57:16.056 INFO:tasks.rados.rados.0.smithi079.stdout:467: snap_remove snap 32 2024-04-24T01:57:17.035 INFO:tasks.rados.rados.0.smithi079.stdout:467: done (6 left) 2024-04-24T01:57:17.036 INFO:tasks.rados.rados.0.smithi079.stdout:468: delete oid 20 current snap is 58 2024-04-24T01:57:17.036 INFO:tasks.rados.rados.0.smithi079.stdout:460: expect (ObjNum 188 snap 36 seq_num 188) 2024-04-24T01:57:17.276 INFO:tasks.rados.rados.0.smithi079.stdout:463: expect (ObjNum 165 snap 25 seq_num 165) 2024-04-24T01:57:17.442 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T01:57:17.457 INFO:tasks.rados.rados.0.smithi079.stdout:464: expect (ObjNum 122 snap 14 seq_num 122) 2024-04-24T01:57:17.547 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:17.545+0000 7fab9a59d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:17.551 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:17.550+0000 7fab9a59d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:17.575 INFO:tasks.rados.rados.0.smithi079.stdout:466: expect (ObjNum 220 snap 49 seq_num 220) 2024-04-24T01:57:17.848 INFO:tasks.rados.rados.0.smithi079.stdout:465: finishing write tid 1 to smithi07929762-24 2024-04-24T01:57:17.848 INFO:tasks.rados.rados.0.smithi079.stdout:465: finishing write tid 2 to smithi07929762-24 2024-04-24T01:57:17.848 INFO:tasks.rados.rados.0.smithi079.stdout:460: done (6 left) 2024-04-24T01:57:17.848 INFO:tasks.rados.rados.0.smithi079.stdout:461: done (5 left) 2024-04-24T01:57:17.848 INFO:tasks.rados.rados.0.smithi079.stdout:463: done (4 left) 2024-04-24T01:57:17.849 INFO:tasks.rados.rados.0.smithi079.stdout:464: done (3 left) 2024-04-24T01:57:17.849 INFO:tasks.rados.rados.0.smithi079.stdout:466: done (2 left) 2024-04-24T01:57:17.849 INFO:tasks.rados.rados.0.smithi079.stdout:468: done (1 left) 2024-04-24T01:57:17.849 INFO:tasks.rados.rados.0.smithi079.stdout:469: read oid 49 snap 55 2024-04-24T01:57:17.849 INFO:tasks.rados.rados.0.smithi079.stdout:469: expect (ObjNum 143 snap 18 seq_num 143) 2024-04-24T01:57:17.849 INFO:tasks.rados.rados.0.smithi079.stdout:470: snap_create 2024-04-24T01:57:17.849 INFO:tasks.rados.rados.0.smithi079.stdout:465: finishing write tid 3 to smithi07929762-24 2024-04-24T01:57:17.849 INFO:tasks.rados.rados.0.smithi079.stdout:465: finishing write tid 4 to smithi07929762-24 2024-04-24T01:57:17.849 INFO:tasks.rados.rados.0.smithi079.stdout:465: finishing write tid 5 to smithi07929762-24 2024-04-24T01:57:17.849 INFO:tasks.rados.rados.0.smithi079.stdout:465: oid 24 updating version 0 to 86 2024-04-24T01:57:17.849 INFO:tasks.rados.rados.0.smithi079.stdout:465: oid 24 version 86 is already newer than 83 2024-04-24T01:57:17.849 INFO:tasks.rados.rados.0.smithi079.stdout:465: oid 24 version 86 is already newer than 86 2024-04-24T01:57:17.849 INFO:tasks.rados.rados.0.smithi079.stdout:465: oid 24 version 86 is already newer than 86 2024-04-24T01:57:17.850 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 24 v 86 (ObjNum 234 snap 58 seq_num 234) dirty exists 2024-04-24T01:57:17.850 INFO:tasks.rados.rados.0.smithi079.stdout:465: left oid 24 (ObjNum 234 snap 58 seq_num 234) 2024-04-24T01:57:17.946 INFO:tasks.rados.rados.0.smithi079.stdout:469: expect (ObjNum 143 snap 18 seq_num 143) 2024-04-24T01:57:18.159 INFO:tasks.rados.rados.0.smithi079.stdout:465: done (2 left) 2024-04-24T01:57:18.159 INFO:tasks.rados.rados.0.smithi079.stdout:469: done (1 left) 2024-04-24T01:57:18.159 INFO:tasks.rados.rados.0.smithi079.stdout:470: done (0 left) 2024-04-24T01:57:18.159 INFO:tasks.rados.rados.0.smithi079.stdout:471: delete oid 12 current snap is 59 2024-04-24T01:57:18.202 INFO:tasks.rados.rados.0.smithi079.stdout:471: done (0 left) 2024-04-24T01:57:18.202 INFO:tasks.rados.rados.0.smithi079.stdout:472: snap_remove snap 52 2024-04-24T01:57:19.164 INFO:tasks.rados.rados.0.smithi079.stdout:472: done (0 left) 2024-04-24T01:57:19.165 INFO:tasks.rados.rados.0.smithi079.stdout:473: snap_create 2024-04-24T01:57:19.643 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T01:57:19.757 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:19.755+0000 7f561c0ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:19.761 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:19.760+0000 7f561c0ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:20.164 INFO:tasks.rados.rados.0.smithi079.stdout:473: done (0 left) 2024-04-24T01:57:20.164 INFO:tasks.rados.rados.0.smithi079.stdout:474: snap_remove snap 47 2024-04-24T01:57:20.421 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-04-24T01:57:20.421 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-04-24T01:57:20.422 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T01:57:20.530 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:20.526+0000 7fe33dced640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:20.534 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:20.533+0000 7fe33dced640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:20.722 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:57:20.723 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-24T01:57:20.733 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":180,"stamp":"2024-04-24T01:57:20.162948+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":129499959,"num_objects":70,"num_object_clones":25,"num_object_copies":210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":1034,"num_read_kb":621892,"num_write":1000,"num_write_kb":243520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1336,"ondisk_log_size":1336,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":452720,"kb_used_data":328784,"kb_used_omap":0,"kb_used_meta":123904,"kb_avail":377034640,"statfs":{"total":386547056640,"available":386083471360,"internally_reserved":0,"allocated":336674816,"data_stored":370992292,"data_compressed":89889,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":126877696},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-2218524,"num_objects":-10,"num_object_clones":-9,"num_object_copies":-30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":41,"num_read_kb":40909,"num_write":11,"num_write_kb":3501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.136530"},"pg_stats":[{"pgid":"3.f","version":"123'65","reported_seq":405,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-24T01:57:18.241281+0000","last_change":"2024-04-24T01:57:17.947575+0000","last_active":"2024-04-24T01:57:18.241281+0000","last_peered":"2024-04-24T01:57:18.241281+0000","last_clean":"2024-04-24T01:57:18.241281+0000","last_became_active":"2024-04-24T01:55:30.386726+0000","last_became_peered":"2024-04-24T01:55:30.386726+0000","last_unstale":"2024-04-24T01:57:18.241281+0000","last_undegraded":"2024-04-24T01:57:18.241281+0000","last_fullsized":"2024-04-24T01:57:18.241281+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0,"stat_sum":{"num_bytes":7159782,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":54,"num_read_kb":46559,"num_write":37,"num_write_kb":11452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"123'113","reported_seq":453,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-24T01:57:18.185745+0000","last_change":"2024-04-24T01:57:17.160184+0000","last_active":"2024-04-24T01:57:18.185745+0000","last_peered":"2024-04-24T01:57:18.185745+0000","last_clean":"2024-04-24T01:57:18.185745+0000","last_became_active":"2024-04-24T01:55:30.321866+0000","last_became_peered":"2024-04-24T01:55:30.321866+0000","last_unstale":"2024-04-24T01:57:18.185745+0000","last_undegraded":"2024-04-24T01:57:18.185745+0000","last_fullsized":"2024-04-24T01:57:18.185745+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":8,"snaptrim_duration":0,"stat_sum":{"num_bytes":11798481,"num_objects":6,"num_object_clones":2,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":80,"num_read_kb":47296,"num_write":76,"num_write_kb":21303,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"123'121","reported_seq":437,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-04-24T01:57:17.148387+0000","last_change":"2024-04-24T01:57:17.174765+0000","last_active":"2024-04-24T01:57:17.148387+0000","last_peered":"2024-04-24T01:57:17.148387+0000","last_clean":"2024-04-24T01:57:17.148387+0000","last_became_active":"2024-04-24T01:55:30.320186+0000","last_became_peered":"2024-04-24T01:55:30.320186+0000","last_unstale":"2024-04-24T01:57:17.148387+0000","last_undegraded":"2024-04-24T01:57:17.148387+0000","last_fullsized":"2024-04-24T01:57:17.148387+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":10,"snaptrim_duration":0,"stat_sum":{"num_bytes":11126234,"num_objects":6,"num_object_clones":3,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":49,"num_read_kb":16608,"num_write":83,"num_write_kb":23135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"123'90","reported_seq":430,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-24T01:57:18.239658+0000","last_change":"2024-04-24T01:57:17.938559+0000","last_active":"2024-04-24T01:57:18.239658+0000","last_peered":"2024-04-24T01:57:18.239658+0000","last_clean":"2024-04-24T01:57:18.239658+0000","last_became_active":"2024-04-24T01:55:30.288081+0000","last_became_peered":"2024-04-24T01:55:30.288081+0000","last_unstale":"2024-04-24T01:57:18.239658+0000","last_undegraded":"2024-04-24T01:57:18.239658+0000","last_fullsized":"2024-04-24T01:57:18.239658+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":9,"snaptrim_duration":0,"stat_sum":{"num_bytes":9227043,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":74,"num_read_kb":54174,"num_write":51,"num_write_kb":14960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"123'46","reported_seq":384,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-24T01:57:18.253068+0000","last_change":"2024-04-24T01:57:18.209154+0000","last_active":"2024-04-24T01:57:18.253068+0000","last_peered":"2024-04-24T01:57:18.253068+0000","last_clean":"2024-04-24T01:57:18.253068+0000","last_became_active":"2024-04-24T01:55:30.333815+0000","last_became_peered":"2024-04-24T01:55:30.333815+0000","last_unstale":"2024-04-24T01:57:18.253068+0000","last_undegraded":"2024-04-24T01:57:18.253068+0000","last_fullsized":"2024-04-24T01:57:18.253068+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0,"stat_sum":{"num_bytes":3798382,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":32,"num_read_kb":17556,"num_write":36,"num_write_kb":10196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"123'142","reported_seq":498,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-24T01:57:18.234726+0000","last_change":"2024-04-24T01:57:17.948367+0000","last_active":"2024-04-24T01:57:18.234726+0000","last_peered":"2024-04-24T01:57:18.234726+0000","last_clean":"2024-04-24T01:57:18.234726+0000","last_became_active":"2024-04-24T01:55:30.315075+0000","last_became_peered":"2024-04-24T01:55:30.315075+0000","last_unstale":"2024-04-24T01:57:18.234726+0000","last_undegraded":"2024-04-24T01:57:18.234726+0000","last_fullsized":"2024-04-24T01:57:18.234726+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":16,"snaptrim_duration":0,"stat_sum":{"num_bytes":17580122,"num_objects":8,"num_object_clones":2,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":116,"num_read_kb":68328,"num_write":108,"num_write_kb":29911,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"112'37","reported_seq":370,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-24T01:57:18.243659+0000","last_change":"2024-04-24T01:57:17.879354+0000","last_active":"2024-04-24T01:57:18.243659+0000","last_peered":"2024-04-24T01:57:18.243659+0000","last_clean":"2024-04-24T01:57:18.243659+0000","last_became_active":"2024-04-24T01:55:30.350300+0000","last_became_peered":"2024-04-24T01:55:30.350300+0000","last_unstale":"2024-04-24T01:57:18.243659+0000","last_undegraded":"2024-04-24T01:57:18.243659+0000","last_fullsized":"2024-04-24T01:57:18.243659+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0,"stat_sum":{"num_bytes":3462242,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":17,"num_read_kb":5006,"num_write":31,"num_write_kb":7547,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"97'7","reported_seq":328,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-24T01:57:17.062751+0000","last_change":"2024-04-24T01:57:17.072587+0000","last_active":"2024-04-24T01:57:17.062751+0000","last_peered":"2024-04-24T01:57:17.062751+0000","last_clean":"2024-04-24T01:57:17.062751+0000","last_became_active":"2024-04-24T01:55:30.376813+0000","last_became_peered":"2024-04-24T01:55:30.376813+0000","last_unstale":"2024-04-24T01:57:17.062751+0000","last_undegraded":"2024-04-24T01:57:17.062751+0000","last_fullsized":"2024-04-24T01:57:17.062751+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'97","reported_seq":428,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-24T01:57:18.248635+0000","last_change":"2024-04-24T01:55:16.196470+0000","last_active":"2024-04-24T01:57:18.248635+0000","last_peered":"2024-04-24T01:57:18.248635+0000","last_clean":"2024-04-24T01:57:18.248635+0000","last_became_active":"2024-04-24T01:55:16.190464+0000","last_became_peered":"2024-04-24T01:55:16.190464+0000","last_unstale":"2024-04-24T01:57:18.248635+0000","last_undegraded":"2024-04-24T01:57:18.248635+0000","last_fullsized":"2024-04-24T01:57:18.248635+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:15.127115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:15.127115+0000","last_clean_scrub_stamp":"2024-04-24T01:55:15.127115+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":286,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-04-24T01:57:17.065627+0000","last_change":"2024-04-24T01:55:18.236148+0000","last_active":"2024-04-24T01:57:17.065627+0000","last_peered":"2024-04-24T01:57:17.065627+0000","last_clean":"2024-04-24T01:57:17.065627+0000","last_became_active":"2024-04-24T01:55:18.230154+0000","last_became_peered":"2024-04-24T01:55:18.230154+0000","last_unstale":"2024-04-24T01:57:17.065627+0000","last_undegraded":"2024-04-24T01:57:17.065627+0000","last_fullsized":"2024-04-24T01:57:17.065627+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"123'122","reported_seq":472,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-24T01:57:17.082568+0000","last_change":"2024-04-24T01:57:17.145062+0000","last_active":"2024-04-24T01:57:17.082568+0000","last_peered":"2024-04-24T01:57:17.082568+0000","last_clean":"2024-04-24T01:57:17.082568+0000","last_became_active":"2024-04-24T01:55:30.316222+0000","last_became_peered":"2024-04-24T01:55:30.316222+0000","last_unstale":"2024-04-24T01:57:17.082568+0000","last_undegraded":"2024-04-24T01:57:17.082568+0000","last_fullsized":"2024-04-24T01:57:17.082568+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":10,"snaptrim_duration":0,"stat_sum":{"num_bytes":12487601,"num_objects":7,"num_object_clones":3,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":108,"num_read_kb":71509,"num_write":90,"num_write_kb":25806,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":295,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-24T01:57:18.248258+0000","last_change":"2024-04-24T01:55:18.265905+0000","last_active":"2024-04-24T01:57:18.248258+0000","last_peered":"2024-04-24T01:57:18.248258+0000","last_clean":"2024-04-24T01:57:18.248258+0000","last_became_active":"2024-04-24T01:55:18.257020+0000","last_became_peered":"2024-04-24T01:55:18.257020+0000","last_unstale":"2024-04-24T01:57:18.248258+0000","last_undegraded":"2024-04-24T01:57:18.248258+0000","last_fullsized":"2024-04-24T01:57:18.248258+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"123'92","reported_seq":435,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-24T01:57:18.248008+0000","last_change":"2024-04-24T01:57:18.248008+0000","last_active":"2024-04-24T01:57:18.248008+0000","last_peered":"2024-04-24T01:57:18.248008+0000","last_clean":"2024-04-24T01:57:18.248008+0000","last_became_active":"2024-04-24T01:55:30.344129+0000","last_became_peered":"2024-04-24T01:55:30.344129+0000","last_unstale":"2024-04-24T01:57:18.248008+0000","last_undegraded":"2024-04-24T01:57:18.248008+0000","last_fullsized":"2024-04-24T01:57:18.248008+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0,"stat_sum":{"num_bytes":10437147,"num_objects":6,"num_object_clones":3,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":72,"num_read_kb":47561,"num_write":60,"num_write_kb":17763,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":286,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-04-24T01:57:17.057912+0000","last_change":"2024-04-24T01:55:18.215251+0000","last_active":"2024-04-24T01:57:17.057912+0000","last_peered":"2024-04-24T01:57:17.057912+0000","last_clean":"2024-04-24T01:57:17.057912+0000","last_became_active":"2024-04-24T01:55:18.206605+0000","last_became_peered":"2024-04-24T01:55:18.206605+0000","last_unstale":"2024-04-24T01:57:17.057912+0000","last_undegraded":"2024-04-24T01:57:17.057912+0000","last_fullsized":"2024-04-24T01:57:17.057912+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"123'38","reported_seq":376,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-24T01:57:18.257155+0000","last_change":"2024-04-24T01:57:18.216636+0000","last_active":"2024-04-24T01:57:18.257155+0000","last_peered":"2024-04-24T01:57:18.257155+0000","last_clean":"2024-04-24T01:57:18.257155+0000","last_became_active":"2024-04-24T01:55:30.401366+0000","last_became_peered":"2024-04-24T01:55:30.401366+0000","last_unstale":"2024-04-24T01:57:18.257155+0000","last_undegraded":"2024-04-24T01:57:18.257155+0000","last_fullsized":"2024-04-24T01:57:18.257155+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":3,"snaptrim_duration":0,"stat_sum":{"num_bytes":7176556,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":32,"num_read_kb":25812,"num_write":24,"num_write_kb":6694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":295,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-24T01:57:18.246354+0000","last_change":"2024-04-24T01:55:18.230893+0000","last_active":"2024-04-24T01:57:18.246354+0000","last_peered":"2024-04-24T01:57:18.246354+0000","last_clean":"2024-04-24T01:57:18.246354+0000","last_became_active":"2024-04-24T01:55:18.225188+0000","last_became_peered":"2024-04-24T01:55:18.225188+0000","last_unstale":"2024-04-24T01:57:18.246354+0000","last_undegraded":"2024-04-24T01:57:18.246354+0000","last_fullsized":"2024-04-24T01:57:18.246354+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"118'68","reported_seq":411,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-24T01:57:18.179922+0000","last_change":"2024-04-24T01:57:17.056970+0000","last_active":"2024-04-24T01:57:18.179922+0000","last_peered":"2024-04-24T01:57:18.179922+0000","last_clean":"2024-04-24T01:57:18.179922+0000","last_became_active":"2024-04-24T01:55:30.343318+0000","last_became_peered":"2024-04-24T01:55:30.343318+0000","last_unstale":"2024-04-24T01:57:18.179922+0000","last_undegraded":"2024-04-24T01:57:18.179922+0000","last_fullsized":"2024-04-24T01:57:18.179922+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":7,"snaptrim_duration":0,"stat_sum":{"num_bytes":10453921,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":61,"num_read_kb":57810,"num_write":51,"num_write_kb":15645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"123'95","reported_seq":441,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-24T01:57:17.100447+0000","last_change":"2024-04-24T01:57:17.169806+0000","last_active":"2024-04-24T01:57:17.100447+0000","last_peered":"2024-04-24T01:57:17.100447+0000","last_clean":"2024-04-24T01:57:17.100447+0000","last_became_active":"2024-04-24T01:55:30.345308+0000","last_became_peered":"2024-04-24T01:55:30.345308+0000","last_unstale":"2024-04-24T01:57:17.100447+0000","last_undegraded":"2024-04-24T01:57:17.100447+0000","last_fullsized":"2024-04-24T01:57:17.100447+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":9,"snaptrim_duration":0,"stat_sum":{"num_bytes":8185009,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":97,"num_read_kb":66852,"num_write":65,"num_write_kb":16455,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":291,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-24T01:57:18.187052+0000","last_change":"2024-04-24T01:55:18.224238+0000","last_active":"2024-04-24T01:57:18.187052+0000","last_peered":"2024-04-24T01:57:18.187052+0000","last_clean":"2024-04-24T01:57:18.187052+0000","last_became_active":"2024-04-24T01:55:18.219131+0000","last_became_peered":"2024-04-24T01:55:18.219131+0000","last_unstale":"2024-04-24T01:57:18.187052+0000","last_undegraded":"2024-04-24T01:57:18.187052+0000","last_fullsized":"2024-04-24T01:57:18.187052+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"92'43","reported_seq":371,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-24T01:57:18.190808+0000","last_change":"2024-04-24T01:57:17.067042+0000","last_active":"2024-04-24T01:57:18.190808+0000","last_peered":"2024-04-24T01:57:18.190808+0000","last_clean":"2024-04-24T01:57:18.190808+0000","last_became_active":"2024-04-24T01:55:30.319958+0000","last_became_peered":"2024-04-24T01:55:30.319958+0000","last_unstale":"2024-04-24T01:57:18.190808+0000","last_undegraded":"2024-04-24T01:57:18.190808+0000","last_fullsized":"2024-04-24T01:57:18.190808+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":22,"num_read_kb":19606,"num_write":28,"num_write_kb":7720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":295,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-24T01:57:18.257285+0000","last_change":"2024-04-24T01:55:18.235763+0000","last_active":"2024-04-24T01:57:18.257285+0000","last_peered":"2024-04-24T01:57:18.257285+0000","last_clean":"2024-04-24T01:57:18.257285+0000","last_became_active":"2024-04-24T01:55:18.229175+0000","last_became_peered":"2024-04-24T01:55:18.229175+0000","last_unstale":"2024-04-24T01:57:18.257285+0000","last_undegraded":"2024-04-24T01:57:18.257285+0000","last_fullsized":"2024-04-24T01:57:18.257285+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"112'32","reported_seq":376,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-24T01:57:18.241977+0000","last_change":"2024-04-24T01:57:18.153116+0000","last_active":"2024-04-24T01:57:18.241977+0000","last_peered":"2024-04-24T01:57:18.241977+0000","last_clean":"2024-04-24T01:57:18.241977+0000","last_became_active":"2024-04-24T01:55:30.372690+0000","last_became_peered":"2024-04-24T01:55:30.372690+0000","last_unstale":"2024-04-24T01:57:18.241977+0000","last_undegraded":"2024-04-24T01:57:18.241977+0000","last_fullsized":"2024-04-24T01:57:18.241977+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":3,"snaptrim_duration":0,"stat_sum":{"num_bytes":6269011,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":36,"num_read_kb":24088,"num_write":29,"num_write_kb":8522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":291,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-24T01:57:18.191678+0000","last_change":"2024-04-24T01:55:18.278351+0000","last_active":"2024-04-24T01:57:18.191678+0000","last_peered":"2024-04-24T01:57:18.191678+0000","last_clean":"2024-04-24T01:57:18.191678+0000","last_became_active":"2024-04-24T01:55:18.273668+0000","last_became_peered":"2024-04-24T01:55:18.273668+0000","last_unstale":"2024-04-24T01:57:18.191678+0000","last_undegraded":"2024-04-24T01:57:18.191678+0000","last_fullsized":"2024-04-24T01:57:18.191678+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"123'128","reported_seq":463,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-24T01:57:18.193911+0000","last_change":"2024-04-24T01:57:17.144132+0000","last_active":"2024-04-24T01:57:18.193911+0000","last_peered":"2024-04-24T01:57:18.193911+0000","last_clean":"2024-04-24T01:57:18.193911+0000","last_became_active":"2024-04-24T01:55:30.363123+0000","last_became_peered":"2024-04-24T01:55:30.363123+0000","last_unstale":"2024-04-24T01:57:18.193911+0000","last_undegraded":"2024-04-24T01:57:18.193911+0000","last_fullsized":"2024-04-24T01:57:18.193911+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":9,"snaptrim_duration":0,"stat_sum":{"num_bytes":9748060,"num_objects":8,"num_object_clones":4,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":79,"num_read_kb":53035,"num_write":89,"num_write_kb":23669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":286,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-24T01:57:17.061070+0000","last_change":"2024-04-24T01:55:18.241117+0000","last_active":"2024-04-24T01:57:17.061070+0000","last_peered":"2024-04-24T01:57:17.061070+0000","last_clean":"2024-04-24T01:57:17.061070+0000","last_became_active":"2024-04-24T01:55:18.234313+0000","last_became_peered":"2024-04-24T01:55:18.234313+0000","last_unstale":"2024-04-24T01:57:17.061070+0000","last_undegraded":"2024-04-24T01:57:17.061070+0000","last_fullsized":"2024-04-24T01:57:17.061070+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":128909591,"num_objects":68,"num_object_clones":25,"num_object_copies":204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":931,"num_read_kb":621801,"num_write":863,"num_write_kb":241626,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":330416128,"data_stored":365361344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1239,"ondisk_log_size":1239,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":1}],"osd_stats":[{"osd":3,"up_from":10,"seq":42949673084,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":124136,"kb_used_data":93152,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94247704,"statfs":{"total":96636764160,"available":96509648896,"internally_reserved":0,"allocated":95387648,"data_stored":106374138,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673084,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":94076,"kb_used_data":63092,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94277764,"statfs":{"total":96636764160,"available":96540430336,"internally_reserved":0,"allocated":64606208,"data_stored":70531615,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705789,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":108064,"kb_used_data":77080,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94263776,"statfs":{"total":96636764160,"available":96526106624,"internally_reserved":0,"allocated":78929920,"data_stored":86258150,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705788,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":126444,"kb_used_data":95460,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94245396,"statfs":{"total":96636764160,"available":96507285504,"internally_reserved":0,"allocated":97751040,"data_stored":107828389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":76963840,"data_stored":84248128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":95621120,"data_stored":106249079,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":63524864,"data_stored":69510807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":94306304,"data_stored":105353330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T01:57:21.165 INFO:tasks.rados.rados.0.smithi079.stdout:474: done (0 left) 2024-04-24T01:57:21.165 INFO:tasks.rados.rados.0.smithi079.stdout:475: read oid 6 snap -1 2024-04-24T01:57:21.165 INFO:tasks.rados.rados.0.smithi079.stdout:475: expect (ObjNum 163 snap 24 seq_num 163) 2024-04-24T01:57:21.165 INFO:tasks.rados.rados.0.smithi079.stdout:476: snap_remove snap 55 2024-04-24T01:57:21.858 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T01:57:21.965 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:21.962+0000 7fd94066d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:21.968 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:21.968+0000 7fd94066d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:22.166 INFO:tasks.rados.rados.0.smithi079.stdout:476: done (1 left) 2024-04-24T01:57:22.167 INFO:tasks.rados.rados.0.smithi079.stdout:477: read oid 25 snap 46 2024-04-24T01:57:22.167 INFO:tasks.rados.rados.0.smithi079.stdout:477: expect (ObjNum 137 snap 17 seq_num 137) 2024-04-24T01:57:22.167 INFO:tasks.rados.rados.0.smithi079.stdout:478: delete oid 2 current snap is 60 2024-04-24T01:57:22.167 INFO:tasks.rados.rados.0.smithi079.stdout:475: expect (ObjNum 163 snap 24 seq_num 163) 2024-04-24T01:57:22.366 INFO:tasks.rados.rados.0.smithi079.stdout:477: expect (ObjNum 137 snap 17 seq_num 137) 2024-04-24T01:57:22.505 INFO:tasks.rados.rados.0.smithi079.stdout:475: done (2 left) 2024-04-24T01:57:22.505 INFO:tasks.rados.rados.0.smithi079.stdout:477: done (1 left) 2024-04-24T01:57:22.505 INFO:tasks.rados.rados.0.smithi079.stdout:478: done (0 left) 2024-04-24T01:57:22.505 INFO:tasks.rados.rados.0.smithi079.stdout:479: read oid 16 snap -1 2024-04-24T01:57:22.505 INFO:tasks.rados.rados.0.smithi079.stdout:479: expect (ObjNum 176 snap 33 seq_num 176) 2024-04-24T01:57:22.505 INFO:tasks.rados.rados.0.smithi079.stdout:480: snap_create 2024-04-24T01:57:22.524 INFO:tasks.rados.rados.0.smithi079.stdout:479: expect (ObjNum 176 snap 33 seq_num 176) 2024-04-24T01:57:23.176 INFO:tasks.rados.rados.0.smithi079.stdout:479: done (1 left) 2024-04-24T01:57:23.176 INFO:tasks.rados.rados.0.smithi079.stdout:480: done (0 left) 2024-04-24T01:57:23.176 INFO:tasks.rados.rados.0.smithi079.stdout:481: snap_create 2024-04-24T01:57:24.063 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T01:57:24.177 INFO:tasks.rados.rados.0.smithi079.stdout:481: done (0 left) 2024-04-24T01:57:24.177 INFO:tasks.rados.rados.0.smithi079.stdout:482: snap_remove snap 61 2024-04-24T01:57:24.202 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:24.199+0000 7f6e2e26d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:24.206 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:24.205+0000 7f6e2e26d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:25.181 INFO:tasks.rados.rados.0.smithi079.stdout:482: done (0 left) 2024-04-24T01:57:25.186 INFO:tasks.rados.rados.0.smithi079.stdout:483: delete oid 48 current snap is 62 2024-04-24T01:57:25.267 INFO:tasks.rados.rados.0.smithi079.stdout:483: done (0 left) 2024-04-24T01:57:25.267 INFO:tasks.rados.rados.0.smithi079.stdout:484: read oid 46 snap -1 2024-04-24T01:57:25.268 INFO:tasks.rados.rados.0.smithi079.stdout:484: expect deleted 2024-04-24T01:57:25.268 INFO:tasks.rados.rados.0.smithi079.stdout:485: write oid 40 current snap is 62 2024-04-24T01:57:25.268 INFO:tasks.rados.rados.0.smithi079.stdout:485: seq_num 239 ranges {416873=791454,1919314=97559} 2024-04-24T01:57:25.273 INFO:tasks.rados.rados.0.smithi079.stdout:485: writing smithi07929762-40 from 416873 to 1208327 tid 1 2024-04-24T01:57:25.278 INFO:tasks.rados.rados.0.smithi079.stdout:485: writing smithi07929762-40 from 1919314 to 2016873 tid 2 2024-04-24T01:57:25.278 INFO:tasks.rados.rados.0.smithi079.stdout:486: snap_create 2024-04-24T01:57:25.345 INFO:tasks.rados.rados.0.smithi079.stdout:485: finishing write tid 1 to smithi07929762-40 2024-04-24T01:57:25.352 INFO:tasks.rados.rados.0.smithi079.stdout:485: finishing write tid 2 to smithi07929762-40 2024-04-24T01:57:25.357 INFO:tasks.rados.rados.0.smithi079.stdout:485: finishing write tid 3 to smithi07929762-40 2024-04-24T01:57:25.358 INFO:tasks.rados.rados.0.smithi079.stdout:485: finishing write tid 4 to smithi07929762-40 2024-04-24T01:57:25.358 INFO:tasks.rados.rados.0.smithi079.stdout:485: oid 40 updating version 0 to 103 2024-04-24T01:57:25.358 INFO:tasks.rados.rados.0.smithi079.stdout:485: oid 40 version 103 is already newer than 103 2024-04-24T01:57:25.359 INFO:tasks.rados.rados.0.smithi079.stdout:485: oid 40 version 103 is already newer than 103 2024-04-24T01:57:25.359 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 40 v 103 (ObjNum 239 snap 62 seq_num 239) dirty exists 2024-04-24T01:57:25.359 INFO:tasks.rados.rados.0.smithi079.stdout:485: left oid 40 (ObjNum 239 snap 62 seq_num 239) 2024-04-24T01:57:25.735 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-04-24T01:57:25.735 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-04-24T01:57:25.736 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T01:57:25.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:25.839+0000 7f6688114640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:25.847 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:25.847+0000 7f6688114640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:26.031 INFO:tasks.rados.rados.0.smithi079.stdout:484: done (2 left) 2024-04-24T01:57:26.031 INFO:tasks.rados.rados.0.smithi079.stdout:485: done (1 left) 2024-04-24T01:57:26.031 INFO:tasks.rados.rados.0.smithi079.stdout:486: done (0 left) 2024-04-24T01:57:26.032 INFO:tasks.rados.rados.0.smithi079.stdout:487: snap_create 2024-04-24T01:57:26.039 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:57:26.039 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-24T01:57:26.050 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":189,"stamp":"2024-04-24T01:57:26.030083+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":129499959,"num_objects":69,"num_object_clones":24,"num_object_copies":207,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":1052,"num_read_kb":639365,"num_write":1000,"num_write_kb":243520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1356,"ondisk_log_size":1356,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":441520,"kb_used_data":317584,"kb_used_omap":0,"kb_used_meta":123904,"kb_avail":377045840,"statfs":{"total":386547056640,"available":386094940160,"internally_reserved":0,"allocated":325206016,"data_stored":356796776,"data_compressed":89889,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":126877696},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":-1,"num_object_clones":-1,"num_object_copies":-3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":17473,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.865340"},"pg_stats":[{"pgid":"3.f","version":"123'65","reported_seq":419,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-24T01:57:23.186014+0000","last_change":"2024-04-24T01:57:22.319677+0000","last_active":"2024-04-24T01:57:23.186014+0000","last_peered":"2024-04-24T01:57:23.186014+0000","last_clean":"2024-04-24T01:57:23.186014+0000","last_became_active":"2024-04-24T01:55:30.386726+0000","last_became_peered":"2024-04-24T01:55:30.386726+0000","last_unstale":"2024-04-24T01:57:23.186014+0000","last_undegraded":"2024-04-24T01:57:23.186014+0000","last_fullsized":"2024-04-24T01:57:23.186014+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0,"stat_sum":{"num_bytes":7159782,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":54,"num_read_kb":46559,"num_write":37,"num_write_kb":11452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"128'117","reported_seq":468,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-24T01:57:24.153107+0000","last_change":"2024-04-24T01:57:23.237109+0000","last_active":"2024-04-24T01:57:24.153107+0000","last_peered":"2024-04-24T01:57:24.153107+0000","last_clean":"2024-04-24T01:57:24.153107+0000","last_became_active":"2024-04-24T01:55:30.321866+0000","last_became_peered":"2024-04-24T01:55:30.321866+0000","last_unstale":"2024-04-24T01:57:24.153107+0000","last_undegraded":"2024-04-24T01:57:24.153107+0000","last_fullsized":"2024-04-24T01:57:24.153107+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":8,"snaptrim_duration":0,"stat_sum":{"num_bytes":11798481,"num_objects":6,"num_object_clones":2,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":80,"num_read_kb":47296,"num_write":76,"num_write_kb":21303,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"123'121","reported_seq":452,"reported_epoch":128,"state":"active+clean+snaptrim","last_fresh":"2024-04-24T01:57:22.191422+0000","last_change":"2024-04-24T01:57:22.191096+0000","last_active":"2024-04-24T01:57:22.191422+0000","last_peered":"2024-04-24T01:57:22.191422+0000","last_clean":"2024-04-24T01:57:22.191422+0000","last_became_active":"2024-04-24T01:55:30.320186+0000","last_became_peered":"2024-04-24T01:55:30.320186+0000","last_unstale":"2024-04-24T01:57:22.191422+0000","last_undegraded":"2024-04-24T01:57:22.191422+0000","last_fullsized":"2024-04-24T01:57:22.191422+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":10,"snaptrim_duration":0,"stat_sum":{"num_bytes":11126234,"num_objects":6,"num_object_clones":3,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":49,"num_read_kb":16608,"num_write":83,"num_write_kb":23135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"127'92","reported_seq":445,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-24T01:57:23.189855+0000","last_change":"2024-04-24T01:57:22.324908+0000","last_active":"2024-04-24T01:57:23.189855+0000","last_peered":"2024-04-24T01:57:23.189855+0000","last_clean":"2024-04-24T01:57:23.189855+0000","last_became_active":"2024-04-24T01:55:30.288081+0000","last_became_peered":"2024-04-24T01:55:30.288081+0000","last_unstale":"2024-04-24T01:57:23.189855+0000","last_undegraded":"2024-04-24T01:57:23.189855+0000","last_fullsized":"2024-04-24T01:57:23.189855+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":9,"snaptrim_duration":0,"stat_sum":{"num_bytes":9227043,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":74,"num_read_kb":54174,"num_write":51,"num_write_kb":14960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"128'48","reported_seq":394,"reported_epoch":129,"state":"active+clean+snaptrim","last_fresh":"2024-04-24T01:57:24.163980+0000","last_change":"2024-04-24T01:57:22.227010+0000","last_active":"2024-04-24T01:57:24.163980+0000","last_peered":"2024-04-24T01:57:24.163980+0000","last_clean":"2024-04-24T01:57:24.163980+0000","last_became_active":"2024-04-24T01:55:30.333815+0000","last_became_peered":"2024-04-24T01:55:30.333815+0000","last_unstale":"2024-04-24T01:57:24.163980+0000","last_undegraded":"2024-04-24T01:57:24.163980+0000","last_fullsized":"2024-04-24T01:57:24.163980+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0,"stat_sum":{"num_bytes":3798382,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":32,"num_read_kb":17556,"num_write":36,"num_write_kb":10196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"127'144","reported_seq":516,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-24T01:57:23.187131+0000","last_change":"2024-04-24T01:57:22.324683+0000","last_active":"2024-04-24T01:57:23.187131+0000","last_peered":"2024-04-24T01:57:23.187131+0000","last_clean":"2024-04-24T01:57:23.187131+0000","last_became_active":"2024-04-24T01:55:30.315075+0000","last_became_peered":"2024-04-24T01:55:30.315075+0000","last_unstale":"2024-04-24T01:57:23.187131+0000","last_undegraded":"2024-04-24T01:57:23.187131+0000","last_fullsized":"2024-04-24T01:57:23.187131+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":144,"log_dups_size":0,"ondisk_log_size":144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":16,"snaptrim_duration":0,"stat_sum":{"num_bytes":17580122,"num_objects":8,"num_object_clones":2,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":122,"num_read_kb":73781,"num_write":108,"num_write_kb":29911,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"112'37","reported_seq":384,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-24T01:57:23.199392+0000","last_change":"2024-04-24T01:57:22.328741+0000","last_active":"2024-04-24T01:57:23.199392+0000","last_peered":"2024-04-24T01:57:23.199392+0000","last_clean":"2024-04-24T01:57:23.199392+0000","last_became_active":"2024-04-24T01:55:30.350300+0000","last_became_peered":"2024-04-24T01:55:30.350300+0000","last_unstale":"2024-04-24T01:57:23.199392+0000","last_undegraded":"2024-04-24T01:57:23.199392+0000","last_fullsized":"2024-04-24T01:57:23.199392+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0,"stat_sum":{"num_bytes":3462242,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":17,"num_read_kb":5006,"num_write":31,"num_write_kb":7547,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"97'7","reported_seq":344,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-24T01:57:22.213519+0000","last_change":"2024-04-24T01:57:22.222123+0000","last_active":"2024-04-24T01:57:22.213519+0000","last_peered":"2024-04-24T01:57:22.213519+0000","last_clean":"2024-04-24T01:57:22.213519+0000","last_became_active":"2024-04-24T01:55:30.376813+0000","last_became_peered":"2024-04-24T01:55:30.376813+0000","last_unstale":"2024-04-24T01:57:22.213519+0000","last_undegraded":"2024-04-24T01:57:22.213519+0000","last_fullsized":"2024-04-24T01:57:22.213519+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'97","reported_seq":437,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-24T01:57:24.154853+0000","last_change":"2024-04-24T01:55:16.196470+0000","last_active":"2024-04-24T01:57:24.154853+0000","last_peered":"2024-04-24T01:57:24.154853+0000","last_clean":"2024-04-24T01:57:24.154853+0000","last_became_active":"2024-04-24T01:55:16.190464+0000","last_became_peered":"2024-04-24T01:55:16.190464+0000","last_unstale":"2024-04-24T01:57:24.154853+0000","last_undegraded":"2024-04-24T01:57:24.154853+0000","last_fullsized":"2024-04-24T01:57:24.154853+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:15.127115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:15.127115+0000","last_clean_scrub_stamp":"2024-04-24T01:55:15.127115+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":299,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-24T01:57:22.200388+0000","last_change":"2024-04-24T01:55:18.236148+0000","last_active":"2024-04-24T01:57:22.200388+0000","last_peered":"2024-04-24T01:57:22.200388+0000","last_clean":"2024-04-24T01:57:22.200388+0000","last_became_active":"2024-04-24T01:55:18.230154+0000","last_became_peered":"2024-04-24T01:55:18.230154+0000","last_unstale":"2024-04-24T01:57:22.200388+0000","last_undegraded":"2024-04-24T01:57:22.200388+0000","last_fullsized":"2024-04-24T01:57:22.200388+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"128'126","reported_seq":491,"reported_epoch":128,"state":"active+clean+snaptrim","last_fresh":"2024-04-24T01:57:22.230772+0000","last_change":"2024-04-24T01:57:22.202290+0000","last_active":"2024-04-24T01:57:22.230772+0000","last_peered":"2024-04-24T01:57:22.230772+0000","last_clean":"2024-04-24T01:57:22.230772+0000","last_became_active":"2024-04-24T01:55:30.316222+0000","last_became_peered":"2024-04-24T01:55:30.316222+0000","last_unstale":"2024-04-24T01:57:22.230772+0000","last_undegraded":"2024-04-24T01:57:22.230772+0000","last_fullsized":"2024-04-24T01:57:22.230772+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":10,"snaptrim_duration":0,"stat_sum":{"num_bytes":12487601,"num_objects":7,"num_object_clones":3,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":114,"num_read_kb":76676,"num_write":90,"num_write_kb":25806,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":306,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-24T01:57:23.196857+0000","last_change":"2024-04-24T01:55:18.265905+0000","last_active":"2024-04-24T01:57:23.196857+0000","last_peered":"2024-04-24T01:57:23.196857+0000","last_clean":"2024-04-24T01:57:23.196857+0000","last_became_active":"2024-04-24T01:55:18.257020+0000","last_became_peered":"2024-04-24T01:55:18.257020+0000","last_unstale":"2024-04-24T01:57:23.196857+0000","last_undegraded":"2024-04-24T01:57:23.196857+0000","last_fullsized":"2024-04-24T01:57:23.196857+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"123'92","reported_seq":442,"reported_epoch":129,"state":"active+clean+snaptrim","last_fresh":"2024-04-24T01:57:22.214298+0000","last_change":"2024-04-24T01:57:22.213842+0000","last_active":"2024-04-24T01:57:22.214298+0000","last_peered":"2024-04-24T01:57:22.214298+0000","last_clean":"2024-04-24T01:57:22.214298+0000","last_became_active":"2024-04-24T01:55:30.344129+0000","last_became_peered":"2024-04-24T01:55:30.344129+0000","last_unstale":"2024-04-24T01:57:22.214298+0000","last_undegraded":"2024-04-24T01:57:22.214298+0000","last_fullsized":"2024-04-24T01:57:22.214298+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0,"stat_sum":{"num_bytes":10437147,"num_objects":6,"num_object_clones":3,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":72,"num_read_kb":47561,"num_write":60,"num_write_kb":17763,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":299,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-24T01:57:22.186242+0000","last_change":"2024-04-24T01:55:18.215251+0000","last_active":"2024-04-24T01:57:22.186242+0000","last_peered":"2024-04-24T01:57:22.186242+0000","last_clean":"2024-04-24T01:57:22.186242+0000","last_became_active":"2024-04-24T01:55:18.206605+0000","last_became_peered":"2024-04-24T01:55:18.206605+0000","last_unstale":"2024-04-24T01:57:22.186242+0000","last_undegraded":"2024-04-24T01:57:22.186242+0000","last_fullsized":"2024-04-24T01:57:22.186242+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"123'38","reported_seq":385,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-24T01:57:22.243379+0000","last_change":"2024-04-24T01:57:22.248229+0000","last_active":"2024-04-24T01:57:22.243379+0000","last_peered":"2024-04-24T01:57:22.243379+0000","last_clean":"2024-04-24T01:57:22.243379+0000","last_became_active":"2024-04-24T01:55:30.401366+0000","last_became_peered":"2024-04-24T01:55:30.401366+0000","last_unstale":"2024-04-24T01:57:22.243379+0000","last_undegraded":"2024-04-24T01:57:22.243379+0000","last_fullsized":"2024-04-24T01:57:22.243379+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":3,"snaptrim_duration":0,"stat_sum":{"num_bytes":7176556,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":32,"num_read_kb":25812,"num_write":24,"num_write_kb":6694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":306,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-24T01:57:23.198204+0000","last_change":"2024-04-24T01:55:18.230893+0000","last_active":"2024-04-24T01:57:23.198204+0000","last_peered":"2024-04-24T01:57:23.198204+0000","last_clean":"2024-04-24T01:57:23.198204+0000","last_became_active":"2024-04-24T01:55:18.225188+0000","last_became_peered":"2024-04-24T01:55:18.225188+0000","last_unstale":"2024-04-24T01:57:23.198204+0000","last_undegraded":"2024-04-24T01:57:23.198204+0000","last_fullsized":"2024-04-24T01:57:23.198204+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"118'68","reported_seq":422,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-24T01:57:22.188559+0000","last_change":"2024-04-24T01:57:22.197758+0000","last_active":"2024-04-24T01:57:22.188559+0000","last_peered":"2024-04-24T01:57:22.188559+0000","last_clean":"2024-04-24T01:57:22.188559+0000","last_became_active":"2024-04-24T01:55:30.343318+0000","last_became_peered":"2024-04-24T01:55:30.343318+0000","last_unstale":"2024-04-24T01:57:22.188559+0000","last_undegraded":"2024-04-24T01:57:22.188559+0000","last_fullsized":"2024-04-24T01:57:22.188559+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":7,"snaptrim_duration":0,"stat_sum":{"num_bytes":10453921,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":61,"num_read_kb":57810,"num_write":51,"num_write_kb":15645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"125'97","reported_seq":457,"reported_epoch":128,"state":"active+clean+snaptrim","last_fresh":"2024-04-24T01:57:22.201601+0000","last_change":"2024-04-24T01:57:22.201191+0000","last_active":"2024-04-24T01:57:22.201601+0000","last_peered":"2024-04-24T01:57:22.201601+0000","last_clean":"2024-04-24T01:57:22.201601+0000","last_became_active":"2024-04-24T01:55:30.345308+0000","last_became_peered":"2024-04-24T01:55:30.345308+0000","last_unstale":"2024-04-24T01:57:22.201601+0000","last_undegraded":"2024-04-24T01:57:22.201601+0000","last_fullsized":"2024-04-24T01:57:22.201601+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":9,"snaptrim_duration":0,"stat_sum":{"num_bytes":8185009,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":97,"num_read_kb":66852,"num_write":65,"num_write_kb":16455,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":300,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-24T01:57:22.195918+0000","last_change":"2024-04-24T01:55:18.224238+0000","last_active":"2024-04-24T01:57:22.195918+0000","last_peered":"2024-04-24T01:57:22.195918+0000","last_clean":"2024-04-24T01:57:22.195918+0000","last_became_active":"2024-04-24T01:55:18.219131+0000","last_became_peered":"2024-04-24T01:55:18.219131+0000","last_unstale":"2024-04-24T01:57:22.195918+0000","last_undegraded":"2024-04-24T01:57:22.195918+0000","last_fullsized":"2024-04-24T01:57:22.195918+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"92'43","reported_seq":382,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-24T01:57:22.197236+0000","last_change":"2024-04-24T01:57:22.205416+0000","last_active":"2024-04-24T01:57:22.197236+0000","last_peered":"2024-04-24T01:57:22.197236+0000","last_clean":"2024-04-24T01:57:22.197236+0000","last_became_active":"2024-04-24T01:55:30.319958+0000","last_became_peered":"2024-04-24T01:55:30.319958+0000","last_unstale":"2024-04-24T01:57:22.197236+0000","last_undegraded":"2024-04-24T01:57:22.197236+0000","last_fullsized":"2024-04-24T01:57:22.197236+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":22,"num_read_kb":19606,"num_write":28,"num_write_kb":7720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":304,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-24T01:57:24.165953+0000","last_change":"2024-04-24T01:55:18.235763+0000","last_active":"2024-04-24T01:57:24.165953+0000","last_peered":"2024-04-24T01:57:24.165953+0000","last_clean":"2024-04-24T01:57:24.165953+0000","last_became_active":"2024-04-24T01:55:18.229175+0000","last_became_peered":"2024-04-24T01:55:18.229175+0000","last_unstale":"2024-04-24T01:57:24.165953+0000","last_undegraded":"2024-04-24T01:57:24.165953+0000","last_fullsized":"2024-04-24T01:57:24.165953+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"112'32","reported_seq":388,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-24T01:57:24.154024+0000","last_change":"2024-04-24T01:57:22.256841+0000","last_active":"2024-04-24T01:57:24.154024+0000","last_peered":"2024-04-24T01:57:24.154024+0000","last_clean":"2024-04-24T01:57:24.154024+0000","last_became_active":"2024-04-24T01:55:30.372690+0000","last_became_peered":"2024-04-24T01:55:30.372690+0000","last_unstale":"2024-04-24T01:57:24.154024+0000","last_undegraded":"2024-04-24T01:57:24.154024+0000","last_fullsized":"2024-04-24T01:57:24.154024+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":3,"snaptrim_duration":0,"stat_sum":{"num_bytes":6269011,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":36,"num_read_kb":24088,"num_write":29,"num_write_kb":8522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":299,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-24T01:57:22.195532+0000","last_change":"2024-04-24T01:55:18.278351+0000","last_active":"2024-04-24T01:57:22.195532+0000","last_peered":"2024-04-24T01:57:22.195532+0000","last_clean":"2024-04-24T01:57:22.195532+0000","last_became_active":"2024-04-24T01:55:18.273668+0000","last_became_peered":"2024-04-24T01:55:18.273668+0000","last_unstale":"2024-04-24T01:57:22.195532+0000","last_undegraded":"2024-04-24T01:57:22.195532+0000","last_fullsized":"2024-04-24T01:57:22.195532+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"127'132","reported_seq":479,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-24T01:57:22.210860+0000","last_change":"2024-04-24T01:57:22.210860+0000","last_active":"2024-04-24T01:57:22.210860+0000","last_peered":"2024-04-24T01:57:22.210860+0000","last_clean":"2024-04-24T01:57:22.210860+0000","last_became_active":"2024-04-24T01:55:30.363123+0000","last_became_peered":"2024-04-24T01:55:30.363123+0000","last_unstale":"2024-04-24T01:57:22.210860+0000","last_undegraded":"2024-04-24T01:57:22.210860+0000","last_fullsized":"2024-04-24T01:57:22.210860+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":132,"log_dups_size":0,"ondisk_log_size":132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":10,"snaptrim_duration":0,"stat_sum":{"num_bytes":9748060,"num_objects":7,"num_object_clones":3,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":85,"num_read_kb":59888,"num_write":89,"num_write_kb":23669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":299,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-24T01:57:22.195827+0000","last_change":"2024-04-24T01:55:18.241117+0000","last_active":"2024-04-24T01:57:22.195827+0000","last_peered":"2024-04-24T01:57:22.195827+0000","last_clean":"2024-04-24T01:57:22.195827+0000","last_became_active":"2024-04-24T01:55:18.234313+0000","last_became_peered":"2024-04-24T01:55:18.234313+0000","last_unstale":"2024-04-24T01:57:22.195827+0000","last_undegraded":"2024-04-24T01:57:22.195827+0000","last_fullsized":"2024-04-24T01:57:22.195827+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":128909591,"num_objects":67,"num_object_clones":24,"num_object_copies":201,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":949,"num_read_kb":639274,"num_write":863,"num_write_kb":241626,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":320802816,"data_stored":353240787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1259,"ondisk_log_size":1259,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":1}],"osd_stats":[{"osd":3,"up_from":10,"seq":42949673089,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":122568,"kb_used_data":91584,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94249272,"statfs":{"total":96636764160,"available":96511254528,"internally_reserved":0,"allocated":93782016,"data_stored":104358717,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673089,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":92508,"kb_used_data":61524,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94279332,"statfs":{"total":96636764160,"available":96542035968,"internally_reserved":0,"allocated":63000576,"data_stored":68516194,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705794,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":107240,"kb_used_data":76256,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94264600,"statfs":{"total":96636764160,"available":96526950400,"internally_reserved":0,"allocated":78086144,"data_stored":85287481,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705793,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":119204,"kb_used_data":88220,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94252636,"statfs":{"total":96636764160,"available":96514699264,"internally_reserved":0,"allocated":90337280,"data_stored":98634384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":76963840,"data_stored":84248128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":89284608,"data_stored":98189084,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61886464,"data_stored":67480526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":92667904,"data_stored":103323049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T01:57:26.305 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T01:57:26.409 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:26.407+0000 7f077e2ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:26.413 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:26.412+0000 7f077e2ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:27.032 INFO:tasks.rados.rados.0.smithi079.stdout:487: done (0 left) 2024-04-24T01:57:27.032 INFO:tasks.rados.rados.0.smithi079.stdout:488: write (excl) oid 9 current snap is 64 2024-04-24T01:57:27.032 INFO:tasks.rados.rados.0.smithi079.stdout:488: seq_num 240 ranges {433680=466703,1577796=455884} 2024-04-24T01:57:27.037 INFO:tasks.rados.rados.0.smithi079.stdout:488: writing smithi07929762-9 from 433680 to 900383 tid 1 2024-04-24T01:57:27.044 INFO:tasks.rados.rados.0.smithi079.stdout:488: writing smithi07929762-9 from 1577796 to 2033680 tid 2 2024-04-24T01:57:27.045 INFO:tasks.rados.rados.0.smithi079.stdout:489: write oid 41 current snap is 64 2024-04-24T01:57:27.045 INFO:tasks.rados.rados.0.smithi079.stdout:489: seq_num 241 ranges {450487=541952,1636278=414209} 2024-04-24T01:57:27.052 INFO:tasks.rados.rados.0.smithi079.stdout:489: writing smithi07929762-41 from 450487 to 992439 tid 1 2024-04-24T01:57:27.063 INFO:tasks.rados.rados.0.smithi079.stdout:489: writing smithi07929762-41 from 1636278 to 2050487 tid 2 2024-04-24T01:57:27.063 INFO:tasks.rados.rados.0.smithi079.stdout:490: snap_create 2024-04-24T01:57:27.115 INFO:tasks.rados.rados.0.smithi079.stdout:488: finishing write tid 1 to smithi07929762-9 2024-04-24T01:57:27.163 INFO:tasks.rados.rados.0.smithi079.stdout:488: finishing write tid 2 to smithi07929762-9 2024-04-24T01:57:27.169 INFO:tasks.rados.rados.0.smithi079.stdout:489: finishing write tid 1 to smithi07929762-41 2024-04-24T01:57:27.174 INFO:tasks.rados.rados.0.smithi079.stdout:489: finishing write tid 2 to smithi07929762-41 2024-04-24T01:57:27.176 INFO:tasks.rados.rados.0.smithi079.stdout:488: finishing write tid 3 to smithi07929762-9 2024-04-24T01:57:27.177 INFO:tasks.rados.rados.0.smithi079.stdout:489: finishing write tid 3 to smithi07929762-41 2024-04-24T01:57:27.178 INFO:tasks.rados.rados.0.smithi079.stdout:488: finishing write tid 4 to smithi07929762-9 2024-04-24T01:57:27.178 INFO:tasks.rados.rados.0.smithi079.stdout:488: oid 9 updating version 0 to 105 2024-04-24T01:57:27.178 INFO:tasks.rados.rados.0.smithi079.stdout:488: oid 9 updating version 105 to 107 2024-04-24T01:57:27.178 INFO:tasks.rados.rados.0.smithi079.stdout:488: oid 9 version 107 is already newer than 107 2024-04-24T01:57:27.178 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 9 v 107 (ObjNum 240 snap 64 seq_num 240) dirty exists 2024-04-24T01:57:27.178 INFO:tasks.rados.rados.0.smithi079.stdout:488: left oid 9 (ObjNum 240 snap 64 seq_num 240) 2024-04-24T01:57:27.178 INFO:tasks.rados.rados.0.smithi079.stdout:489: finishing write tid 4 to smithi07929762-41 2024-04-24T01:57:27.179 INFO:tasks.rados.rados.0.smithi079.stdout:489: oid 41 updating version 0 to 42 2024-04-24T01:57:27.179 INFO:tasks.rados.rados.0.smithi079.stdout:489: oid 41 version 42 is already newer than 42 2024-04-24T01:57:27.179 INFO:tasks.rados.rados.0.smithi079.stdout:489: oid 41 version 42 is already newer than 42 2024-04-24T01:57:27.179 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 41 v 42 (ObjNum 241 snap 64 seq_num 241) dirty exists 2024-04-24T01:57:27.179 INFO:tasks.rados.rados.0.smithi079.stdout:489: left oid 41 (ObjNum 241 snap 64 seq_num 241) 2024-04-24T01:57:28.031 INFO:tasks.rados.rados.0.smithi079.stdout:488: done (2 left) 2024-04-24T01:57:28.031 INFO:tasks.rados.rados.0.smithi079.stdout:489: done (1 left) 2024-04-24T01:57:28.031 INFO:tasks.rados.rados.0.smithi079.stdout:490: done (0 left) 2024-04-24T01:57:28.031 INFO:tasks.rados.rados.0.smithi079.stdout:491: snap_remove snap 59 2024-04-24T01:57:28.506 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T01:57:28.621 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:28.619+0000 7f6036aad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:28.625 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:28.624+0000 7f6036aad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:29.034 INFO:tasks.rados.rados.0.smithi079.stdout:491: done (0 left) 2024-04-24T01:57:29.035 INFO:tasks.rados.rados.0.smithi079.stdout:492: snap_create 2024-04-24T01:57:30.035 INFO:tasks.rados.rados.0.smithi079.stdout:492: done (0 left) 2024-04-24T01:57:30.036 INFO:tasks.rados.rados.0.smithi079.stdout:493: read oid 32 snap -1 2024-04-24T01:57:30.036 INFO:tasks.rados.rados.0.smithi079.stdout:493: expect (ObjNum 209 snap 44 seq_num 209) 2024-04-24T01:57:30.036 INFO:tasks.rados.rados.0.smithi079.stdout:494: write oid 40 current snap is 66 2024-04-24T01:57:30.036 INFO:tasks.rados.rados.0.smithi079.stdout:494: seq_num 242 ranges {467294=617201,1578407=488887} 2024-04-24T01:57:30.041 INFO:tasks.rados.rados.0.smithi079.stdout:494: writing smithi07929762-40 from 467294 to 1084495 tid 1 2024-04-24T01:57:30.049 INFO:tasks.rados.rados.0.smithi079.stdout:494: writing smithi07929762-40 from 1578407 to 2067294 tid 2 2024-04-24T01:57:30.049 INFO:tasks.rados.rados.0.smithi079.stdout:495: snap_create 2024-04-24T01:57:30.131 INFO:tasks.rados.rados.0.smithi079.stdout:494: finishing write tid 1 to smithi07929762-40 2024-04-24T01:57:30.144 INFO:tasks.rados.rados.0.smithi079.stdout:494: finishing write tid 2 to smithi07929762-40 2024-04-24T01:57:30.148 INFO:tasks.rados.rados.0.smithi079.stdout:494: finishing write tid 3 to smithi07929762-40 2024-04-24T01:57:30.149 INFO:tasks.rados.rados.0.smithi079.stdout:493: expect (ObjNum 209 snap 44 seq_num 209) 2024-04-24T01:57:30.336 INFO:tasks.rados.rados.0.smithi079.stdout:494: finishing write tid 4 to smithi07929762-40 2024-04-24T01:57:30.336 INFO:tasks.rados.rados.0.smithi079.stdout:494: oid 40 updating version 0 to 115 2024-04-24T01:57:30.336 INFO:tasks.rados.rados.0.smithi079.stdout:494: oid 40 version 115 is already newer than 115 2024-04-24T01:57:30.336 INFO:tasks.rados.rados.0.smithi079.stdout:494: oid 40 version 115 is already newer than 115 2024-04-24T01:57:30.336 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 40 v 115 (ObjNum 242 snap 66 seq_num 242) dirty exists 2024-04-24T01:57:30.336 INFO:tasks.rados.rados.0.smithi079.stdout:494: left oid 40 (ObjNum 242 snap 66 seq_num 242) 2024-04-24T01:57:30.721 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T01:57:30.827 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:30.825+0000 7f8d3746d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:30.831 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:30.830+0000 7f8d3746d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:31.041 INFO:tasks.rados.rados.0.smithi079.stdout:493: done (2 left) 2024-04-24T01:57:31.041 INFO:tasks.rados.rados.0.smithi079.stdout:494: done (1 left) 2024-04-24T01:57:31.041 INFO:tasks.rados.rados.0.smithi079.stdout:495: done (0 left) 2024-04-24T01:57:31.041 INFO:tasks.rados.rados.0.smithi079.stdout:496: read oid 38 snap -1 2024-04-24T01:57:31.041 INFO:tasks.rados.rados.0.smithi079.stdout:496: expect deleted 2024-04-24T01:57:31.041 INFO:tasks.rados.rados.0.smithi079.stdout:497: snap_remove snap 56 2024-04-24T01:57:31.052 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-04-24T01:57:31.052 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-04-24T01:57:31.053 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T01:57:31.170 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:31.165+0000 7fea9615d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:31.174 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:31.174+0000 7fea9615d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:31.363 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:57:31.364 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-24T01:57:31.376 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":197,"stamp":"2024-04-24T01:57:31.039348+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":123231080,"num_objects":73,"num_object_clones":28,"num_object_copies":219,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":1057,"num_read_kb":639368,"num_write":1013,"num_write_kb":246226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1379,"ondisk_log_size":1379,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":449860,"kb_used_data":325924,"kb_used_omap":0,"kb_used_meta":123904,"kb_avail":377037500,"statfs":{"total":386547056640,"available":386086400000,"internally_reserved":0,"allocated":333746176,"data_stored":369798780,"data_compressed":89889,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":126877696},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-6268879,"num_objects":4,"num_object_clones":4,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":5,"num_read_kb":3,"num_write":13,"num_write_kb":2706,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.008587"},"pg_stats":[{"pgid":"3.f","version":"131'67","reported_seq":428,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-04-24T01:57:26.250586+0000","last_change":"2024-04-24T01:57:25.224820+0000","last_active":"2024-04-24T01:57:26.250586+0000","last_peered":"2024-04-24T01:57:26.250586+0000","last_clean":"2024-04-24T01:57:26.250586+0000","last_became_active":"2024-04-24T01:55:30.386726+0000","last_became_peered":"2024-04-24T01:55:30.386726+0000","last_unstale":"2024-04-24T01:57:26.250586+0000","last_undegraded":"2024-04-24T01:57:26.250586+0000","last_fullsized":"2024-04-24T01:57:26.250586+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0,"stat_sum":{"num_bytes":3512663,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":55,"num_read_kb":46559,"num_write":38,"num_write_kb":11452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"128'117","reported_seq":477,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-04-24T01:57:26.242338+0000","last_change":"2024-04-24T01:57:26.260813+0000","last_active":"2024-04-24T01:57:26.242338+0000","last_peered":"2024-04-24T01:57:26.242338+0000","last_clean":"2024-04-24T01:57:26.242338+0000","last_became_active":"2024-04-24T01:55:30.321866+0000","last_became_peered":"2024-04-24T01:55:30.321866+0000","last_unstale":"2024-04-24T01:57:26.242338+0000","last_undegraded":"2024-04-24T01:57:26.242338+0000","last_fullsized":"2024-04-24T01:57:26.242338+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":8,"snaptrim_duration":0,"stat_sum":{"num_bytes":11798481,"num_objects":6,"num_object_clones":2,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":80,"num_read_kb":47296,"num_write":76,"num_write_kb":21303,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"128'123","reported_seq":470,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-04-24T01:57:27.045369+0000","last_change":"2024-04-24T01:57:25.296314+0000","last_active":"2024-04-24T01:57:27.045369+0000","last_peered":"2024-04-24T01:57:27.045369+0000","last_clean":"2024-04-24T01:57:27.045369+0000","last_became_active":"2024-04-24T01:55:30.320186+0000","last_became_peered":"2024-04-24T01:55:30.320186+0000","last_unstale":"2024-04-24T01:57:27.045369+0000","last_undegraded":"2024-04-24T01:57:27.045369+0000","last_fullsized":"2024-04-24T01:57:27.045369+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":123,"log_dups_size":0,"ondisk_log_size":123,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":10,"snaptrim_duration":0,"stat_sum":{"num_bytes":11126234,"num_objects":6,"num_object_clones":3,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":49,"num_read_kb":16608,"num_write":83,"num_write_kb":23135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"127'92","reported_seq":453,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-04-24T01:57:26.244569+0000","last_change":"2024-04-24T01:57:25.226312+0000","last_active":"2024-04-24T01:57:26.244569+0000","last_peered":"2024-04-24T01:57:26.244569+0000","last_clean":"2024-04-24T01:57:26.244569+0000","last_became_active":"2024-04-24T01:55:30.288081+0000","last_became_peered":"2024-04-24T01:55:30.288081+0000","last_unstale":"2024-04-24T01:57:26.244569+0000","last_undegraded":"2024-04-24T01:57:26.244569+0000","last_fullsized":"2024-04-24T01:57:26.244569+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":9,"snaptrim_duration":0,"stat_sum":{"num_bytes":9227043,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":74,"num_read_kb":54174,"num_write":51,"num_write_kb":14960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"128'48","reported_seq":413,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-24T01:57:28.059662+0000","last_change":"2024-04-24T01:57:25.209294+0000","last_active":"2024-04-24T01:57:28.059662+0000","last_peered":"2024-04-24T01:57:28.059662+0000","last_clean":"2024-04-24T01:57:28.059662+0000","last_became_active":"2024-04-24T01:55:30.333815+0000","last_became_peered":"2024-04-24T01:55:30.333815+0000","last_unstale":"2024-04-24T01:57:28.059662+0000","last_undegraded":"2024-04-24T01:57:28.059662+0000","last_fullsized":"2024-04-24T01:57:28.059662+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0,"stat_sum":{"num_bytes":3798382,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":32,"num_read_kb":17556,"num_write":36,"num_write_kb":10196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"127'144","reported_seq":524,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-04-24T01:57:26.245366+0000","last_change":"2024-04-24T01:57:25.226673+0000","last_active":"2024-04-24T01:57:26.245366+0000","last_peered":"2024-04-24T01:57:26.245366+0000","last_clean":"2024-04-24T01:57:26.245366+0000","last_became_active":"2024-04-24T01:55:30.315075+0000","last_became_peered":"2024-04-24T01:55:30.315075+0000","last_unstale":"2024-04-24T01:57:26.245366+0000","last_undegraded":"2024-04-24T01:57:26.245366+0000","last_fullsized":"2024-04-24T01:57:26.245366+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":144,"log_dups_size":0,"ondisk_log_size":144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":16,"snaptrim_duration":0,"stat_sum":{"num_bytes":17580122,"num_objects":8,"num_object_clones":2,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":122,"num_read_kb":73781,"num_write":108,"num_write_kb":29911,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"112'37","reported_seq":392,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-04-24T01:57:26.253175+0000","last_change":"2024-04-24T01:57:25.230455+0000","last_active":"2024-04-24T01:57:26.253175+0000","last_peered":"2024-04-24T01:57:26.253175+0000","last_clean":"2024-04-24T01:57:26.253175+0000","last_became_active":"2024-04-24T01:55:30.350300+0000","last_became_peered":"2024-04-24T01:55:30.350300+0000","last_unstale":"2024-04-24T01:57:26.253175+0000","last_undegraded":"2024-04-24T01:57:26.253175+0000","last_fullsized":"2024-04-24T01:57:26.253175+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0,"stat_sum":{"num_bytes":3462242,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":17,"num_read_kb":5006,"num_write":31,"num_write_kb":7547,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"97'7","reported_seq":360,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-04-24T01:57:27.052598+0000","last_change":"2024-04-24T01:57:25.307617+0000","last_active":"2024-04-24T01:57:27.052598+0000","last_peered":"2024-04-24T01:57:27.052598+0000","last_clean":"2024-04-24T01:57:27.052598+0000","last_became_active":"2024-04-24T01:55:30.376813+0000","last_became_peered":"2024-04-24T01:55:30.376813+0000","last_unstale":"2024-04-24T01:57:27.052598+0000","last_undegraded":"2024-04-24T01:57:27.052598+0000","last_fullsized":"2024-04-24T01:57:27.052598+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'97","reported_seq":452,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-24T01:57:28.055393+0000","last_change":"2024-04-24T01:55:16.196470+0000","last_active":"2024-04-24T01:57:28.055393+0000","last_peered":"2024-04-24T01:57:28.055393+0000","last_clean":"2024-04-24T01:57:28.055393+0000","last_became_active":"2024-04-24T01:55:16.190464+0000","last_became_peered":"2024-04-24T01:55:16.190464+0000","last_unstale":"2024-04-24T01:57:28.055393+0000","last_undegraded":"2024-04-24T01:57:28.055393+0000","last_fullsized":"2024-04-24T01:57:28.055393+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:15.127115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:15.127115+0000","last_clean_scrub_stamp":"2024-04-24T01:55:15.127115+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":314,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-04-24T01:57:27.057759+0000","last_change":"2024-04-24T01:55:18.236148+0000","last_active":"2024-04-24T01:57:27.057759+0000","last_peered":"2024-04-24T01:57:27.057759+0000","last_clean":"2024-04-24T01:57:27.057759+0000","last_became_active":"2024-04-24T01:55:18.230154+0000","last_became_peered":"2024-04-24T01:55:18.230154+0000","last_unstale":"2024-04-24T01:57:27.057759+0000","last_undegraded":"2024-04-24T01:57:27.057759+0000","last_fullsized":"2024-04-24T01:57:27.057759+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"128'126","reported_seq":509,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-04-24T01:57:27.051624+0000","last_change":"2024-04-24T01:57:25.306400+0000","last_active":"2024-04-24T01:57:27.051624+0000","last_peered":"2024-04-24T01:57:27.051624+0000","last_clean":"2024-04-24T01:57:27.051624+0000","last_became_active":"2024-04-24T01:55:30.316222+0000","last_became_peered":"2024-04-24T01:55:30.316222+0000","last_unstale":"2024-04-24T01:57:27.051624+0000","last_undegraded":"2024-04-24T01:57:27.051624+0000","last_fullsized":"2024-04-24T01:57:27.051624+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":10,"snaptrim_duration":0,"stat_sum":{"num_bytes":12487601,"num_objects":7,"num_object_clones":3,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":114,"num_read_kb":76676,"num_write":90,"num_write_kb":25806,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":313,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-04-24T01:57:26.256352+0000","last_change":"2024-04-24T01:55:18.265905+0000","last_active":"2024-04-24T01:57:26.256352+0000","last_peered":"2024-04-24T01:57:26.256352+0000","last_clean":"2024-04-24T01:57:26.256352+0000","last_became_active":"2024-04-24T01:55:18.257020+0000","last_became_peered":"2024-04-24T01:55:18.257020+0000","last_unstale":"2024-04-24T01:57:26.256352+0000","last_undegraded":"2024-04-24T01:57:26.256352+0000","last_fullsized":"2024-04-24T01:57:26.256352+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"129'96","reported_seq":465,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-24T01:57:28.050832+0000","last_change":"2024-04-24T01:57:25.206514+0000","last_active":"2024-04-24T01:57:28.050832+0000","last_peered":"2024-04-24T01:57:28.050832+0000","last_clean":"2024-04-24T01:57:28.050832+0000","last_became_active":"2024-04-24T01:55:30.344129+0000","last_became_peered":"2024-04-24T01:55:30.344129+0000","last_unstale":"2024-04-24T01:57:28.050832+0000","last_undegraded":"2024-04-24T01:57:28.050832+0000","last_fullsized":"2024-04-24T01:57:28.050832+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0,"stat_sum":{"num_bytes":10437147,"num_objects":6,"num_object_clones":3,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":72,"num_read_kb":47561,"num_write":60,"num_write_kb":17763,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":314,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-04-24T01:57:27.046138+0000","last_change":"2024-04-24T01:55:18.215251+0000","last_active":"2024-04-24T01:57:27.046138+0000","last_peered":"2024-04-24T01:57:27.046138+0000","last_clean":"2024-04-24T01:57:27.046138+0000","last_became_active":"2024-04-24T01:55:18.206605+0000","last_became_peered":"2024-04-24T01:55:18.206605+0000","last_unstale":"2024-04-24T01:57:27.046138+0000","last_undegraded":"2024-04-24T01:57:27.046138+0000","last_fullsized":"2024-04-24T01:57:27.046138+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"133'42","reported_seq":408,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-24T01:57:28.060650+0000","last_change":"2024-04-24T01:57:25.209290+0000","last_active":"2024-04-24T01:57:28.060650+0000","last_peered":"2024-04-24T01:57:28.060650+0000","last_clean":"2024-04-24T01:57:28.060650+0000","last_became_active":"2024-04-24T01:55:30.401366+0000","last_became_peered":"2024-04-24T01:55:30.401366+0000","last_unstale":"2024-04-24T01:57:28.060650+0000","last_undegraded":"2024-04-24T01:57:28.060650+0000","last_fullsized":"2024-04-24T01:57:28.060650+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":3,"snaptrim_duration":0,"stat_sum":{"num_bytes":5327852,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":33,"num_read_kb":25813,"num_write":28,"num_write_kb":7629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":313,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-04-24T01:57:26.254913+0000","last_change":"2024-04-24T01:55:18.230893+0000","last_active":"2024-04-24T01:57:26.254913+0000","last_peered":"2024-04-24T01:57:26.254913+0000","last_clean":"2024-04-24T01:57:26.254913+0000","last_became_active":"2024-04-24T01:55:18.225188+0000","last_became_peered":"2024-04-24T01:55:18.225188+0000","last_unstale":"2024-04-24T01:57:26.254913+0000","last_undegraded":"2024-04-24T01:57:26.254913+0000","last_fullsized":"2024-04-24T01:57:26.254913+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"118'68","reported_seq":433,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-04-24T01:57:26.252435+0000","last_change":"2024-04-24T01:57:26.261343+0000","last_active":"2024-04-24T01:57:26.252435+0000","last_peered":"2024-04-24T01:57:26.252435+0000","last_clean":"2024-04-24T01:57:26.252435+0000","last_became_active":"2024-04-24T01:55:30.343318+0000","last_became_peered":"2024-04-24T01:55:30.343318+0000","last_unstale":"2024-04-24T01:57:26.252435+0000","last_undegraded":"2024-04-24T01:57:26.252435+0000","last_fullsized":"2024-04-24T01:57:26.252435+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":7,"snaptrim_duration":0,"stat_sum":{"num_bytes":10453921,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":61,"num_read_kb":57810,"num_write":51,"num_write_kb":15645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"133'107","reported_seq":483,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-04-24T01:57:27.116256+0000","last_change":"2024-04-24T01:57:25.308522+0000","last_active":"2024-04-24T01:57:27.116256+0000","last_peered":"2024-04-24T01:57:27.116256+0000","last_clean":"2024-04-24T01:57:27.116256+0000","last_became_active":"2024-04-24T01:55:30.345308+0000","last_became_peered":"2024-04-24T01:55:30.345308+0000","last_unstale":"2024-04-24T01:57:27.116256+0000","last_undegraded":"2024-04-24T01:57:27.116256+0000","last_fullsized":"2024-04-24T01:57:27.116256+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":9,"snaptrim_duration":0,"stat_sum":{"num_bytes":7411953,"num_objects":7,"num_object_clones":3,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":100,"num_read_kb":66854,"num_write":73,"num_write_kb":18226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":309,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-04-24T01:57:26.265642+0000","last_change":"2024-04-24T01:55:18.224238+0000","last_active":"2024-04-24T01:57:26.265642+0000","last_peered":"2024-04-24T01:57:26.265642+0000","last_clean":"2024-04-24T01:57:26.265642+0000","last_became_active":"2024-04-24T01:55:18.219131+0000","last_became_peered":"2024-04-24T01:55:18.219131+0000","last_unstale":"2024-04-24T01:57:26.265642+0000","last_undegraded":"2024-04-24T01:57:26.265642+0000","last_fullsized":"2024-04-24T01:57:26.265642+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"128'44","reported_seq":393,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-04-24T01:57:26.270987+0000","last_change":"2024-04-24T01:57:26.278561+0000","last_active":"2024-04-24T01:57:26.270987+0000","last_peered":"2024-04-24T01:57:26.270987+0000","last_clean":"2024-04-24T01:57:26.270987+0000","last_became_active":"2024-04-24T01:55:30.319958+0000","last_became_peered":"2024-04-24T01:55:30.319958+0000","last_unstale":"2024-04-24T01:57:26.270987+0000","last_undegraded":"2024-04-24T01:57:26.270987+0000","last_fullsized":"2024-04-24T01:57:26.270987+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":22,"num_read_kb":19606,"num_write":28,"num_write_kb":7720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":319,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-24T01:57:28.061861+0000","last_change":"2024-04-24T01:55:18.235763+0000","last_active":"2024-04-24T01:57:28.061861+0000","last_peered":"2024-04-24T01:57:28.061861+0000","last_clean":"2024-04-24T01:57:28.061861+0000","last_became_active":"2024-04-24T01:55:18.229175+0000","last_became_peered":"2024-04-24T01:55:18.229175+0000","last_unstale":"2024-04-24T01:57:28.061861+0000","last_undegraded":"2024-04-24T01:57:28.061861+0000","last_fullsized":"2024-04-24T01:57:28.061861+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"112'32","reported_seq":404,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-24T01:57:28.052780+0000","last_change":"2024-04-24T01:57:25.206023+0000","last_active":"2024-04-24T01:57:28.052780+0000","last_peered":"2024-04-24T01:57:28.052780+0000","last_clean":"2024-04-24T01:57:28.052780+0000","last_became_active":"2024-04-24T01:55:30.372690+0000","last_became_peered":"2024-04-24T01:55:30.372690+0000","last_unstale":"2024-04-24T01:57:28.052780+0000","last_undegraded":"2024-04-24T01:57:28.052780+0000","last_fullsized":"2024-04-24T01:57:28.052780+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":3,"snaptrim_duration":0,"stat_sum":{"num_bytes":6269011,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":36,"num_read_kb":24088,"num_write":29,"num_write_kb":8522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":309,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-04-24T01:57:26.272972+0000","last_change":"2024-04-24T01:55:18.278351+0000","last_active":"2024-04-24T01:57:26.272972+0000","last_peered":"2024-04-24T01:57:26.272972+0000","last_clean":"2024-04-24T01:57:26.272972+0000","last_became_active":"2024-04-24T01:55:18.273668+0000","last_became_peered":"2024-04-24T01:55:18.273668+0000","last_unstale":"2024-04-24T01:57:26.272972+0000","last_undegraded":"2024-04-24T01:57:26.272972+0000","last_fullsized":"2024-04-24T01:57:26.272972+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"127'132","reported_seq":490,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-04-24T01:57:26.259186+0000","last_change":"2024-04-24T01:57:26.278782+0000","last_active":"2024-04-24T01:57:26.259186+0000","last_peered":"2024-04-24T01:57:26.259186+0000","last_clean":"2024-04-24T01:57:26.259186+0000","last_became_active":"2024-04-24T01:55:30.363123+0000","last_became_peered":"2024-04-24T01:55:30.363123+0000","last_unstale":"2024-04-24T01:57:26.259186+0000","last_undegraded":"2024-04-24T01:57:26.259186+0000","last_fullsized":"2024-04-24T01:57:26.259186+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":132,"log_dups_size":0,"ondisk_log_size":132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":10,"snaptrim_duration":0,"stat_sum":{"num_bytes":9748060,"num_objects":7,"num_object_clones":3,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":85,"num_read_kb":59888,"num_write":89,"num_write_kb":23669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":314,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-04-24T01:57:27.053551+0000","last_change":"2024-04-24T01:55:18.241117+0000","last_active":"2024-04-24T01:57:27.053551+0000","last_peered":"2024-04-24T01:57:27.053551+0000","last_clean":"2024-04-24T01:57:27.053551+0000","last_became_active":"2024-04-24T01:55:18.234313+0000","last_became_peered":"2024-04-24T01:55:18.234313+0000","last_unstale":"2024-04-24T01:57:27.053551+0000","last_undegraded":"2024-04-24T01:57:27.053551+0000","last_fullsized":"2024-04-24T01:57:27.053551+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":122640712,"num_objects":71,"num_object_clones":28,"num_object_copies":213,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":954,"num_read_kb":639277,"num_write":876,"num_write_kb":244332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":329195520,"data_stored":366176061,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1282,"ondisk_log_size":1282,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":1}],"osd_stats":[{"osd":3,"up_from":10,"seq":42949673094,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":125348,"kb_used_data":94364,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94246492,"statfs":{"total":96636764160,"available":96508407808,"internally_reserved":0,"allocated":96628736,"data_stored":108692705,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673094,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":95272,"kb_used_data":64288,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94276568,"statfs":{"total":96636764160,"available":96539205632,"internally_reserved":0,"allocated":65830912,"data_stored":72842780,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705799,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":107264,"kb_used_data":76280,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94264576,"statfs":{"total":96636764160,"available":96526925824,"internally_reserved":0,"allocated":78110720,"data_stored":85298624,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705798,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":121976,"kb_used_data":90992,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94249864,"statfs":{"total":96636764160,"available":96511860736,"internally_reserved":0,"allocated":93175808,"data_stored":102964671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":76963840,"data_stored":84248128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":92082176,"data_stored":102500842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":64684032,"data_stored":71792284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":95465472,"data_stored":107634807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T01:57:32.042 INFO:tasks.rados.rados.0.smithi079.stdout:497: done (1 left) 2024-04-24T01:57:32.042 INFO:tasks.rados.rados.0.smithi079.stdout:498: write oid 9 current snap is 67 2024-04-24T01:57:32.042 INFO:tasks.rados.rados.0.smithi079.stdout:498: seq_num 243 ranges {484101=692450,1636889=447212} 2024-04-24T01:57:32.048 INFO:tasks.rados.rados.0.smithi079.stdout:498: writing smithi07929762-9 from 484101 to 1176551 tid 1 2024-04-24T01:57:32.055 INFO:tasks.rados.rados.0.smithi079.stdout:498: writing smithi07929762-9 from 1636889 to 2084101 tid 2 2024-04-24T01:57:32.055 INFO:tasks.rados.rados.0.smithi079.stdout:499: write oid 34 current snap is 67 2024-04-24T01:57:32.055 INFO:tasks.rados.rados.0.smithi079.stdout:499: seq_num 244 ranges {500908=484052,1411724=689184} 2024-04-24T01:57:32.061 INFO:tasks.rados.rados.0.smithi079.stdout:499: writing smithi07929762-34 from 500908 to 984960 tid 1 2024-04-24T01:57:32.069 INFO:tasks.rados.rados.0.smithi079.stdout:499: writing smithi07929762-34 from 1411724 to 2100908 tid 2 2024-04-24T01:57:32.069 INFO:tasks.rados.rados.0.smithi079.stdout:500: snap_create 2024-04-24T01:57:32.178 INFO:tasks.rados.rados.0.smithi079.stdout:499: finishing write tid 1 to smithi07929762-34 2024-04-24T01:57:32.178 INFO:tasks.rados.rados.0.smithi079.stdout:498: finishing write tid 1 to smithi07929762-9 2024-04-24T01:57:32.195 INFO:tasks.rados.rados.0.smithi079.stdout:498: finishing write tid 2 to smithi07929762-9 2024-04-24T01:57:32.198 INFO:tasks.rados.rados.0.smithi079.stdout:499: finishing write tid 2 to smithi07929762-34 2024-04-24T01:57:32.201 INFO:tasks.rados.rados.0.smithi079.stdout:498: finishing write tid 3 to smithi07929762-9 2024-04-24T01:57:32.202 INFO:tasks.rados.rados.0.smithi079.stdout:498: finishing write tid 4 to smithi07929762-9 2024-04-24T01:57:32.202 INFO:tasks.rados.rados.0.smithi079.stdout:498: oid 9 updating version 0 to 123 2024-04-24T01:57:32.203 INFO:tasks.rados.rados.0.smithi079.stdout:498: oid 9 version 123 is already newer than 123 2024-04-24T01:57:32.203 INFO:tasks.rados.rados.0.smithi079.stdout:498: oid 9 version 123 is already newer than 123 2024-04-24T01:57:32.203 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 9 v 123 (ObjNum 243 snap 67 seq_num 243) dirty exists 2024-04-24T01:57:32.203 INFO:tasks.rados.rados.0.smithi079.stdout:498: left oid 9 (ObjNum 243 snap 67 seq_num 243) 2024-04-24T01:57:32.209 INFO:tasks.rados.rados.0.smithi079.stdout:499: finishing write tid 3 to smithi07929762-34 2024-04-24T01:57:32.209 INFO:tasks.rados.rados.0.smithi079.stdout:499: finishing write tid 4 to smithi07929762-34 2024-04-24T01:57:32.210 INFO:tasks.rados.rados.0.smithi079.stdout:499: oid 34 updating version 0 to 96 2024-04-24T01:57:32.210 INFO:tasks.rados.rados.0.smithi079.stdout:499: oid 34 version 96 is already newer than 96 2024-04-24T01:57:32.210 INFO:tasks.rados.rados.0.smithi079.stdout:499: oid 34 version 96 is already newer than 96 2024-04-24T01:57:32.210 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 34 v 96 (ObjNum 244 snap 67 seq_num 244) dirty exists 2024-04-24T01:57:32.210 INFO:tasks.rados.rados.0.smithi079.stdout:499: left oid 34 (ObjNum 244 snap 67 seq_num 244) 2024-04-24T01:57:32.925 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T01:57:33.029 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:33.027+0000 7f259c82d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:33.033 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:33.032+0000 7f259c82d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:33.056 INFO:tasks.rados.rados.0.smithi079.stdout:496: done (3 left) 2024-04-24T01:57:33.056 INFO:tasks.rados.rados.0.smithi079.stdout:498: done (2 left) 2024-04-24T01:57:33.056 INFO:tasks.rados.rados.0.smithi079.stdout:499: done (1 left) 2024-04-24T01:57:33.056 INFO:tasks.rados.rados.0.smithi079.stdout:500: done (0 left) 2024-04-24T01:57:33.056 INFO:tasks.rados.rados.0.smithi079.stdout:501: write (excl) oid 44 current snap is 68 2024-04-24T01:57:33.056 INFO:tasks.rados.rados.0.smithi079.stdout:501: seq_num 245 ranges {517715=559301,1870206=247509} 2024-04-24T01:57:33.062 INFO:tasks.rados.rados.0.smithi079.stdout:501: writing smithi07929762-44 from 517715 to 1077016 tid 1 2024-04-24T01:57:33.068 INFO:tasks.rados.rados.0.smithi079.stdout:501: writing smithi07929762-44 from 1870206 to 2117715 tid 2 2024-04-24T01:57:33.068 INFO:tasks.rados.rados.0.smithi079.stdout:502: write oid 49 current snap is 68 2024-04-24T01:57:33.068 INFO:tasks.rados.rados.0.smithi079.stdout:502: seq_num 246 ranges {534522=634550,1812335=322187} 2024-04-24T01:57:33.074 INFO:tasks.rados.rados.0.smithi079.stdout:502: writing smithi07929762-49 from 534522 to 1169072 tid 1 2024-04-24T01:57:33.081 INFO:tasks.rados.rados.0.smithi079.stdout:502: writing smithi07929762-49 from 1812335 to 2134522 tid 2 2024-04-24T01:57:33.081 INFO:tasks.rados.rados.0.smithi079.stdout:503: read oid 17 snap -1 2024-04-24T01:57:33.081 INFO:tasks.rados.rados.0.smithi079.stdout:503: expect (ObjNum 220 snap 49 seq_num 220) 2024-04-24T01:57:33.081 INFO:tasks.rados.rados.0.smithi079.stdout:504: read oid 34 snap -1 2024-04-24T01:57:33.081 INFO:tasks.rados.rados.0.smithi079.stdout:504: expect (ObjNum 244 snap 67 seq_num 244) 2024-04-24T01:57:33.081 INFO:tasks.rados.rados.0.smithi079.stdout:505: read oid 21 snap -1 2024-04-24T01:57:33.081 INFO:tasks.rados.rados.0.smithi079.stdout:505: expect (ObjNum 20 snap 0 seq_num 20) 2024-04-24T01:57:33.081 INFO:tasks.rados.rados.0.smithi079.stdout:506: delete oid 30 current snap is 68 2024-04-24T01:57:33.146 INFO:tasks.rados.rados.0.smithi079.stdout:506: done (5 left) 2024-04-24T01:57:33.147 INFO:tasks.rados.rados.0.smithi079.stdout:507: write (excl) oid 31 current snap is 68 2024-04-24T01:57:33.147 INFO:tasks.rados.rados.0.smithi079.stdout:507: seq_num 248 ranges {568136=785048,1929299=238837} 2024-04-24T01:57:33.154 INFO:tasks.rados.rados.0.smithi079.stdout:507: writing smithi07929762-31 from 568136 to 1353184 tid 1 2024-04-24T01:57:33.160 INFO:tasks.rados.rados.0.smithi079.stdout:507: writing smithi07929762-31 from 1929299 to 2168136 tid 2 2024-04-24T01:57:33.160 INFO:tasks.rados.rados.0.smithi079.stdout:508: write (excl) oid 13 current snap is 68 2024-04-24T01:57:33.160 INFO:tasks.rados.rados.0.smithi079.stdout:503: expect (ObjNum 220 snap 49 seq_num 220) 2024-04-24T01:57:33.464 INFO:tasks.rados.rados.0.smithi079.stdout:505: expect (ObjNum 20 snap 0 seq_num 20) 2024-04-24T01:57:33.552 INFO:tasks.rados.rados.0.smithi079.stdout:502: finishing write tid 1 to smithi07929762-49 2024-04-24T01:57:33.553 INFO:tasks.rados.rados.0.smithi079.stdout:502: finishing write tid 2 to smithi07929762-49 2024-04-24T01:57:33.553 INFO:tasks.rados.rados.0.smithi079.stdout:502: finishing write tid 3 to smithi07929762-49 2024-04-24T01:57:33.553 INFO:tasks.rados.rados.0.smithi079.stdout:504: expect (ObjNum 244 snap 67 seq_num 244) 2024-04-24T01:57:33.653 INFO:tasks.rados.rados.0.smithi079.stdout:502: finishing write tid 4 to smithi07929762-49 2024-04-24T01:57:33.654 INFO:tasks.rados.rados.0.smithi079.stdout:502: oid 49 updating version 0 to 148 2024-04-24T01:57:33.654 INFO:tasks.rados.rados.0.smithi079.stdout:502: oid 49 version 148 is already newer than 148 2024-04-24T01:57:33.654 INFO:tasks.rados.rados.0.smithi079.stdout:502: oid 49 version 148 is already newer than 148 2024-04-24T01:57:33.654 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 49 v 148 (ObjNum 246 snap 68 seq_num 246) dirty exists 2024-04-24T01:57:33.654 INFO:tasks.rados.rados.0.smithi079.stdout:502: left oid 49 (ObjNum 246 snap 68 seq_num 246) 2024-04-24T01:57:33.654 INFO:tasks.rados.rados.0.smithi079.stdout:501: finishing write tid 1 to smithi07929762-44 2024-04-24T01:57:33.654 INFO:tasks.rados.rados.0.smithi079.stdout:508: seq_num 249 ranges {584943=460297,1587781=597162} 2024-04-24T01:57:33.658 INFO:tasks.rados.rados.0.smithi079.stdout:508: writing smithi07929762-13 from 584943 to 1045240 tid 1 2024-04-24T01:57:33.666 INFO:tasks.rados.rados.0.smithi079.stdout:508: writing smithi07929762-13 from 1587781 to 2184943 tid 2 2024-04-24T01:57:33.666 INFO:tasks.rados.rados.0.smithi079.stdout:502: done (6 left) 2024-04-24T01:57:33.666 INFO:tasks.rados.rados.0.smithi079.stdout:503: done (5 left) 2024-04-24T01:57:33.667 INFO:tasks.rados.rados.0.smithi079.stdout:504: done (4 left) 2024-04-24T01:57:33.667 INFO:tasks.rados.rados.0.smithi079.stdout:505: done (3 left) 2024-04-24T01:57:33.667 INFO:tasks.rados.rados.0.smithi079.stdout:509: snap_create 2024-04-24T01:57:33.667 INFO:tasks.rados.rados.0.smithi079.stdout:501: finishing write tid 2 to smithi07929762-44 2024-04-24T01:57:33.667 INFO:tasks.rados.rados.0.smithi079.stdout:507: finishing write tid 1 to smithi07929762-31 2024-04-24T01:57:33.667 INFO:tasks.rados.rados.0.smithi079.stdout:507: finishing write tid 2 to smithi07929762-31 2024-04-24T01:57:33.667 INFO:tasks.rados.rados.0.smithi079.stdout:501: finishing write tid 3 to smithi07929762-44 2024-04-24T01:57:33.667 INFO:tasks.rados.rados.0.smithi079.stdout:501: finishing write tid 4 to smithi07929762-44 2024-04-24T01:57:33.667 INFO:tasks.rados.rados.0.smithi079.stdout:501: oid 44 updating version 0 to 72 2024-04-24T01:57:33.667 INFO:tasks.rados.rados.0.smithi079.stdout:501: oid 44 version 72 is already newer than 72 2024-04-24T01:57:33.667 INFO:tasks.rados.rados.0.smithi079.stdout:501: oid 44 version 72 is already newer than 70 2024-04-24T01:57:33.667 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 44 v 72 (ObjNum 245 snap 68 seq_num 245) dirty exists 2024-04-24T01:57:33.667 INFO:tasks.rados.rados.0.smithi079.stdout:501: left oid 44 (ObjNum 245 snap 68 seq_num 245) 2024-04-24T01:57:33.667 INFO:tasks.rados.rados.0.smithi079.stdout:507: finishing write tid 3 to smithi07929762-31 2024-04-24T01:57:33.667 INFO:tasks.rados.rados.0.smithi079.stdout:507: finishing write tid 4 to smithi07929762-31 2024-04-24T01:57:33.668 INFO:tasks.rados.rados.0.smithi079.stdout:507: oid 31 updating version 0 to 152 2024-04-24T01:57:33.668 INFO:tasks.rados.rados.0.smithi079.stdout:507: oid 31 version 152 is already newer than 152 2024-04-24T01:57:33.668 INFO:tasks.rados.rados.0.smithi079.stdout:507: oid 31 version 152 is already newer than 150 2024-04-24T01:57:33.668 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 31 v 152 (ObjNum 248 snap 68 seq_num 248) dirty exists 2024-04-24T01:57:33.668 INFO:tasks.rados.rados.0.smithi079.stdout:507: left oid 31 (ObjNum 248 snap 68 seq_num 248) 2024-04-24T01:57:33.728 INFO:tasks.rados.rados.0.smithi079.stdout:508: finishing write tid 1 to smithi07929762-13 2024-04-24T01:57:33.744 INFO:tasks.rados.rados.0.smithi079.stdout:508: finishing write tid 2 to smithi07929762-13 2024-04-24T01:57:33.750 INFO:tasks.rados.rados.0.smithi079.stdout:508: finishing write tid 3 to smithi07929762-13 2024-04-24T01:57:33.751 INFO:tasks.rados.rados.0.smithi079.stdout:508: finishing write tid 4 to smithi07929762-13 2024-04-24T01:57:33.751 INFO:tasks.rados.rados.0.smithi079.stdout:508: oid 13 updating version 0 to 130 2024-04-24T01:57:33.751 INFO:tasks.rados.rados.0.smithi079.stdout:508: oid 13 version 130 is already newer than 130 2024-04-24T01:57:33.751 INFO:tasks.rados.rados.0.smithi079.stdout:508: oid 13 version 130 is already newer than 128 2024-04-24T01:57:33.751 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 13 v 130 (ObjNum 249 snap 68 seq_num 249) dirty exists 2024-04-24T01:57:33.751 INFO:tasks.rados.rados.0.smithi079.stdout:508: left oid 13 (ObjNum 249 snap 68 seq_num 249) 2024-04-24T01:57:34.052 INFO:tasks.rados.rados.0.smithi079.stdout:501: done (3 left) 2024-04-24T01:57:34.053 INFO:tasks.rados.rados.0.smithi079.stdout:507: done (2 left) 2024-04-24T01:57:34.053 INFO:tasks.rados.rados.0.smithi079.stdout:508: done (1 left) 2024-04-24T01:57:34.053 INFO:tasks.rados.rados.0.smithi079.stdout:509: done (0 left) 2024-04-24T01:57:34.053 INFO:tasks.rados.rados.0.smithi079.stdout:510: write oid 18 current snap is 69 2024-04-24T01:57:34.053 INFO:tasks.rados.rados.0.smithi079.stdout:510: seq_num 250 ranges {601750=535546,1929910=271840} 2024-04-24T01:57:34.059 INFO:tasks.rados.rados.0.smithi079.stdout:510: writing smithi07929762-18 from 601750 to 1137296 tid 1 2024-04-24T01:57:34.066 INFO:tasks.rados.rados.0.smithi079.stdout:510: writing smithi07929762-18 from 1929910 to 2201750 tid 2 2024-04-24T01:57:34.066 INFO:tasks.rados.rados.0.smithi079.stdout:511: read oid 47 snap -1 2024-04-24T01:57:34.066 INFO:tasks.rados.rados.0.smithi079.stdout:511: expect (ObjNum 193 snap 38 seq_num 193) 2024-04-24T01:57:34.066 INFO:tasks.rados.rados.0.smithi079.stdout:512: snap_remove snap 62 2024-04-24T01:57:35.063 INFO:tasks.rados.rados.0.smithi079.stdout:512: done (2 left) 2024-04-24T01:57:35.063 INFO:tasks.rados.rados.0.smithi079.stdout:513: read oid 41 snap -1 2024-04-24T01:57:35.063 INFO:tasks.rados.rados.0.smithi079.stdout:513: expect (ObjNum 241 snap 64 seq_num 241) 2024-04-24T01:57:35.063 INFO:tasks.rados.rados.0.smithi079.stdout:514: delete oid 31 current snap is 69 2024-04-24T01:57:35.063 INFO:tasks.rados.rados.0.smithi079.stdout:511: expect (ObjNum 193 snap 38 seq_num 193) 2024-04-24T01:57:35.217 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T01:57:35.286 INFO:tasks.rados.rados.0.smithi079.stdout:510: finishing write tid 1 to smithi07929762-18 2024-04-24T01:57:35.286 INFO:tasks.rados.rados.0.smithi079.stdout:510: finishing write tid 2 to smithi07929762-18 2024-04-24T01:57:35.286 INFO:tasks.rados.rados.0.smithi079.stdout:510: finishing write tid 3 to smithi07929762-18 2024-04-24T01:57:35.286 INFO:tasks.rados.rados.0.smithi079.stdout:510: finishing write tid 4 to smithi07929762-18 2024-04-24T01:57:35.286 INFO:tasks.rados.rados.0.smithi079.stdout:510: oid 18 updating version 0 to 156 2024-04-24T01:57:35.286 INFO:tasks.rados.rados.0.smithi079.stdout:510: oid 18 version 156 is already newer than 156 2024-04-24T01:57:35.286 INFO:tasks.rados.rados.0.smithi079.stdout:510: oid 18 version 156 is already newer than 156 2024-04-24T01:57:35.286 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 18 v 156 (ObjNum 250 snap 69 seq_num 250) dirty exists 2024-04-24T01:57:35.286 INFO:tasks.rados.rados.0.smithi079.stdout:510: left oid 18 (ObjNum 250 snap 69 seq_num 250) 2024-04-24T01:57:35.286 INFO:tasks.rados.rados.0.smithi079.stdout:510: done (3 left) 2024-04-24T01:57:35.287 INFO:tasks.rados.rados.0.smithi079.stdout:511: done (2 left) 2024-04-24T01:57:35.287 INFO:tasks.rados.rados.0.smithi079.stdout:514: done (1 left) 2024-04-24T01:57:35.287 INFO:tasks.rados.rados.0.smithi079.stdout:515: write oid 40 current snap is 69 2024-04-24T01:57:35.287 INFO:tasks.rados.rados.0.smithi079.stdout:513: expect (ObjNum 241 snap 64 seq_num 241) 2024-04-24T01:57:35.324 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:35.321+0000 7f1c0695d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:35.327 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:35.327+0000 7f1c0695d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:35.412 INFO:tasks.rados.rados.0.smithi079.stdout:515: seq_num 252 ranges {635364=402397,1763227=472137} 2024-04-24T01:57:35.418 INFO:tasks.rados.rados.0.smithi079.stdout:515: writing smithi07929762-40 from 635364 to 1037761 tid 1 2024-04-24T01:57:35.426 INFO:tasks.rados.rados.0.smithi079.stdout:515: writing smithi07929762-40 from 1763227 to 2235364 tid 2 2024-04-24T01:57:35.426 INFO:tasks.rados.rados.0.smithi079.stdout:513: done (1 left) 2024-04-24T01:57:35.427 INFO:tasks.rados.rados.0.smithi079.stdout:516: snap_create 2024-04-24T01:57:35.454 INFO:tasks.rados.rados.0.smithi079.stdout:515: finishing write tid 1 to smithi07929762-40 2024-04-24T01:57:35.464 INFO:tasks.rados.rados.0.smithi079.stdout:515: finishing write tid 2 to smithi07929762-40 2024-04-24T01:57:35.469 INFO:tasks.rados.rados.0.smithi079.stdout:515: finishing write tid 3 to smithi07929762-40 2024-04-24T01:57:35.470 INFO:tasks.rados.rados.0.smithi079.stdout:515: finishing write tid 4 to smithi07929762-40 2024-04-24T01:57:35.470 INFO:tasks.rados.rados.0.smithi079.stdout:515: oid 40 updating version 0 to 133 2024-04-24T01:57:35.470 INFO:tasks.rados.rados.0.smithi079.stdout:515: oid 40 version 133 is already newer than 133 2024-04-24T01:57:35.470 INFO:tasks.rados.rados.0.smithi079.stdout:515: oid 40 version 133 is already newer than 133 2024-04-24T01:57:35.470 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 40 v 133 (ObjNum 252 snap 69 seq_num 252) dirty exists 2024-04-24T01:57:35.470 INFO:tasks.rados.rados.0.smithi079.stdout:515: left oid 40 (ObjNum 252 snap 69 seq_num 252) 2024-04-24T01:57:36.031 INFO:tasks.rados.rados.0.smithi079.stdout:515: done (1 left) 2024-04-24T01:57:36.031 INFO:tasks.rados.rados.0.smithi079.stdout:516: done (0 left) 2024-04-24T01:57:36.031 INFO:tasks.rados.rados.0.smithi079.stdout:517: snap_remove snap 60 2024-04-24T01:57:36.378 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-04-24T01:57:36.378 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-04-24T01:57:36.378 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-04-24T01:57:36.378 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-04-24T01:57:36.488 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:36.482+0000 7f451bb54640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:36.492 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:36.491+0000 7f451bb54640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:36.507 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T01:57:36.507 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T01:57:36.507 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T01:57:36.507 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T01:57:36.507 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T01:57:36.507 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T01:57:36.507 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T01:57:36.507 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T01:57:36.507 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T01:57:36.507 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T01:57:36.507 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T01:57:36.508 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T01:57:36.510 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T01:57:36.510 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T01:57:36.510 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-04-24T01:57:36.616 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:36.611+0000 7f9f22eed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:36.620 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:36.619+0000 7f9f22eed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:36.634 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T01:57:36.634 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T01:57:36.634 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T01:57:36.634 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T01:57:36.634 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T01:57:36.635 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T01:57:36.635 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T01:57:36.635 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T01:57:36.635 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T01:57:36.635 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T01:57:36.635 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T01:57:36.635 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T01:57:36.637 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T01:57:36.638 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T01:57:36.638 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-04-24T01:57:36.746 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:36.740+0000 7f47d962d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:36.749 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:36.748+0000 7f47d962d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:36.766 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T01:57:36.766 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T01:57:36.767 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T01:57:36.767 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T01:57:36.767 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T01:57:36.767 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T01:57:36.767 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T01:57:36.767 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T01:57:36.767 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T01:57:36.767 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T01:57:36.767 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T01:57:36.767 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T01:57:36.769 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T01:57:36.770 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T01:57:36.770 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-04-24T01:57:36.878 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:36.872+0000 7f6b3479d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:36.882 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:36.881+0000 7f6b3479d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:36.899 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T01:57:36.899 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T01:57:36.899 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T01:57:36.899 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T01:57:36.899 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T01:57:36.899 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T01:57:36.899 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T01:57:36.899 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T01:57:36.900 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T01:57:36.900 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T01:57:36.900 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T01:57:36.900 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T01:57:36.902 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T01:57:36.902 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T01:57:37.033 INFO:tasks.rados.rados.0.smithi079.stdout:517: done (0 left) 2024-04-24T01:57:37.034 INFO:tasks.rados.rados.0.smithi079.stdout:518: snap_remove snap 39 2024-04-24T01:57:37.424 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T01:57:37.529 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:37.526+0000 7fcade91d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:37.532 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:37.532+0000 7fcade91d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:38.035 INFO:tasks.rados.rados.0.smithi079.stdout:518: done (0 left) 2024-04-24T01:57:38.035 INFO:tasks.rados.rados.0.smithi079.stdout:519: write (excl) oid 22 current snap is 70 2024-04-24T01:57:38.035 INFO:tasks.rados.rados.0.smithi079.stdout:519: seq_num 253 ranges {652171=477646,1821709=430462} 2024-04-24T01:57:38.042 INFO:tasks.rados.rados.0.smithi079.stdout:519: writing smithi07929762-22 from 652171 to 1129817 tid 1 2024-04-24T01:57:38.049 INFO:tasks.rados.rados.0.smithi079.stdout:519: writing smithi07929762-22 from 1821709 to 2252171 tid 2 2024-04-24T01:57:38.049 INFO:tasks.rados.rados.0.smithi079.stdout:520: read oid 23 snap 50 2024-04-24T01:57:38.049 INFO:tasks.rados.rados.0.smithi079.stdout:520: expect (ObjNum 206 snap 44 seq_num 206) 2024-04-24T01:57:38.049 INFO:tasks.rados.rados.0.smithi079.stdout:521: read oid 16 snap -1 2024-04-24T01:57:38.050 INFO:tasks.rados.rados.0.smithi079.stdout:521: expect (ObjNum 176 snap 33 seq_num 176) 2024-04-24T01:57:38.050 INFO:tasks.rados.rados.0.smithi079.stdout:522: read oid 1 snap -1 2024-04-24T01:57:38.050 INFO:tasks.rados.rados.0.smithi079.stdout:522: expect (ObjNum 133 snap 15 seq_num 133) 2024-04-24T01:57:38.050 INFO:tasks.rados.rados.0.smithi079.stdout:523: delete oid 45 current snap is 70 2024-04-24T01:57:38.152 INFO:tasks.rados.rados.0.smithi079.stdout:520: expect (ObjNum 206 snap 44 seq_num 206) 2024-04-24T01:57:38.300 INFO:tasks.rados.rados.0.smithi079.stdout:521: expect (ObjNum 176 snap 33 seq_num 176) 2024-04-24T01:57:38.422 INFO:tasks.rados.rados.0.smithi079.stdout:519: finishing write tid 1 to smithi07929762-22 2024-04-24T01:57:38.422 INFO:tasks.rados.rados.0.smithi079.stdout:522: expect (ObjNum 133 snap 15 seq_num 133) 2024-04-24T01:57:38.587 INFO:tasks.rados.rados.0.smithi079.stdout:519: finishing write tid 2 to smithi07929762-22 2024-04-24T01:57:38.587 INFO:tasks.rados.rados.0.smithi079.stdout:519: finishing write tid 3 to smithi07929762-22 2024-04-24T01:57:38.587 INFO:tasks.rados.rados.0.smithi079.stdout:519: finishing write tid 4 to smithi07929762-22 2024-04-24T01:57:38.587 INFO:tasks.rados.rados.0.smithi079.stdout:519: oid 22 updating version 0 to 54 2024-04-24T01:57:38.587 INFO:tasks.rados.rados.0.smithi079.stdout:519: oid 22 updating version 54 to 56 2024-04-24T01:57:38.587 INFO:tasks.rados.rados.0.smithi079.stdout:519: oid 22 version 56 is already newer than 56 2024-04-24T01:57:38.587 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 22 v 56 (ObjNum 253 snap 70 seq_num 253) dirty exists 2024-04-24T01:57:38.588 INFO:tasks.rados.rados.0.smithi079.stdout:519: left oid 22 (ObjNum 253 snap 70 seq_num 253) 2024-04-24T01:57:38.588 INFO:tasks.rados.rados.0.smithi079.stdout:519: done (4 left) 2024-04-24T01:57:38.588 INFO:tasks.rados.rados.0.smithi079.stdout:520: done (3 left) 2024-04-24T01:57:38.588 INFO:tasks.rados.rados.0.smithi079.stdout:521: done (2 left) 2024-04-24T01:57:38.588 INFO:tasks.rados.rados.0.smithi079.stdout:522: done (1 left) 2024-04-24T01:57:38.588 INFO:tasks.rados.rados.0.smithi079.stdout:523: done (0 left) 2024-04-24T01:57:38.588 INFO:tasks.rados.rados.0.smithi079.stdout:524: write (excl) oid 18 current snap is 70 2024-04-24T01:57:38.588 INFO:tasks.rados.rados.0.smithi079.stdout:524: seq_num 255 ranges {685785=628144,1822320=463465} 2024-04-24T01:57:38.593 INFO:tasks.rados.rados.0.smithi079.stdout:524: writing smithi07929762-18 from 685785 to 1313929 tid 1 2024-04-24T01:57:38.600 INFO:tasks.rados.rados.0.smithi079.stdout:524: writing smithi07929762-18 from 1822320 to 2285785 tid 2 2024-04-24T01:57:38.601 INFO:tasks.rados.rados.0.smithi079.stdout:525: snap_remove snap 65 2024-04-24T01:57:39.052 INFO:tasks.rados.rados.0.smithi079.stdout:525: done (1 left) 2024-04-24T01:57:39.052 INFO:tasks.rados.rados.0.smithi079.stdout:526: read oid 41 snap 66 2024-04-24T01:57:39.052 INFO:tasks.rados.rados.0.smithi079.stdout:526: expect (ObjNum 241 snap 64 seq_num 241) 2024-04-24T01:57:39.052 INFO:tasks.rados.rados.0.smithi079.stdout:524: finishing write tid 1 to smithi07929762-18 2024-04-24T01:57:39.053 INFO:tasks.rados.rados.0.smithi079.stdout:527: read oid 15 snap -1 2024-04-24T01:57:39.053 INFO:tasks.rados.rados.0.smithi079.stdout:527: expect (ObjNum 197 snap 40 seq_num 197) 2024-04-24T01:57:39.053 INFO:tasks.rados.rados.0.smithi079.stdout:528: read oid 7 snap 46 2024-04-24T01:57:39.053 INFO:tasks.rados.rados.0.smithi079.stdout:528: expect (ObjNum 167 snap 26 seq_num 167) 2024-04-24T01:57:39.053 INFO:tasks.rados.rados.0.smithi079.stdout:529: write oid 26 current snap is 70 2024-04-24T01:57:39.053 INFO:tasks.rados.rados.0.smithi079.stdout:524: finishing write tid 2 to smithi07929762-18 2024-04-24T01:57:39.053 INFO:tasks.rados.rados.0.smithi079.stdout:524: finishing write tid 3 to smithi07929762-18 2024-04-24T01:57:39.053 INFO:tasks.rados.rados.0.smithi079.stdout:524: finishing write tid 4 to smithi07929762-18 2024-04-24T01:57:39.053 INFO:tasks.rados.rados.0.smithi079.stdout:524: oid 18 updating version 0 to 178 2024-04-24T01:57:39.053 INFO:tasks.rados.rados.0.smithi079.stdout:524: oid 18 updating version 178 to 180 2024-04-24T01:57:39.053 INFO:tasks.rados.rados.0.smithi079.stdout:524: oid 18 version 180 is already newer than 180 2024-04-24T01:57:39.054 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 18 v 180 (ObjNum 255 snap 70 seq_num 255) dirty exists 2024-04-24T01:57:39.054 INFO:tasks.rados.rados.0.smithi079.stdout:524: left oid 18 (ObjNum 255 snap 70 seq_num 255) 2024-04-24T01:57:39.054 INFO:tasks.rados.rados.0.smithi079.stdout:529: seq_num 256 ranges {702592=703393,1880802=421790} 2024-04-24T01:57:39.058 INFO:tasks.rados.rados.0.smithi079.stdout:529: writing smithi07929762-26 from 702592 to 1405985 tid 1 2024-04-24T01:57:39.064 INFO:tasks.rados.rados.0.smithi079.stdout:529: writing smithi07929762-26 from 1880802 to 2302592 tid 2 2024-04-24T01:57:39.064 INFO:tasks.rados.rados.0.smithi079.stdout:524: done (4 left) 2024-04-24T01:57:39.064 INFO:tasks.rados.rados.0.smithi079.stdout:530: snap_remove snap 64 2024-04-24T01:57:39.626 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T01:57:39.731 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:39.729+0000 7fb302464640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:39.735 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:39.734+0000 7fb302464640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:40.055 INFO:tasks.rados.rados.0.smithi079.stdout:530: done (4 left) 2024-04-24T01:57:40.056 INFO:tasks.rados.rados.0.smithi079.stdout:531: write oid 30 current snap is 70 2024-04-24T01:57:40.056 INFO:tasks.rados.rados.0.smithi079.stdout:531: seq_num 257 ranges {719399=778642,1939284=380115} 2024-04-24T01:57:40.063 INFO:tasks.rados.rados.0.smithi079.stdout:531: writing smithi07929762-30 from 719399 to 1498041 tid 1 2024-04-24T01:57:40.068 INFO:tasks.rados.rados.0.smithi079.stdout:531: writing smithi07929762-30 from 1939284 to 2319399 tid 2 2024-04-24T01:57:40.068 INFO:tasks.rados.rados.0.smithi079.stdout:532: read oid 50 snap 54 2024-04-24T01:57:40.068 INFO:tasks.rados.rados.0.smithi079.stdout:532: expect (ObjNum 162 snap 24 seq_num 162) 2024-04-24T01:57:40.068 INFO:tasks.rados.rados.0.smithi079.stdout:533: delete oid 13 current snap is 70 2024-04-24T01:57:40.068 INFO:tasks.rados.rados.0.smithi079.stdout:527: expect (ObjNum 197 snap 40 seq_num 197) 2024-04-24T01:57:40.223 INFO:tasks.rados.rados.0.smithi079.stdout:526: expect (ObjNum 241 snap 64 seq_num 241) 2024-04-24T01:57:40.339 INFO:tasks.rados.rados.0.smithi079.stdout:528: expect (ObjNum 167 snap 26 seq_num 167) 2024-04-24T01:57:40.504 INFO:tasks.rados.rados.0.smithi079.stdout:529: finishing write tid 1 to smithi07929762-26 2024-04-24T01:57:40.504 INFO:tasks.rados.rados.0.smithi079.stdout:529: finishing write tid 2 to smithi07929762-26 2024-04-24T01:57:40.504 INFO:tasks.rados.rados.0.smithi079.stdout:529: finishing write tid 3 to smithi07929762-26 2024-04-24T01:57:40.505 INFO:tasks.rados.rados.0.smithi079.stdout:529: finishing write tid 4 to smithi07929762-26 2024-04-24T01:57:40.505 INFO:tasks.rados.rados.0.smithi079.stdout:529: oid 26 updating version 0 to 142 2024-04-24T01:57:40.505 INFO:tasks.rados.rados.0.smithi079.stdout:529: oid 26 version 142 is already newer than 142 2024-04-24T01:57:40.505 INFO:tasks.rados.rados.0.smithi079.stdout:529: oid 26 version 142 is already newer than 142 2024-04-24T01:57:40.505 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 26 v 142 (ObjNum 256 snap 70 seq_num 256) dirty exists 2024-04-24T01:57:40.505 INFO:tasks.rados.rados.0.smithi079.stdout:529: left oid 26 (ObjNum 256 snap 70 seq_num 256) 2024-04-24T01:57:40.505 INFO:tasks.rados.rados.0.smithi079.stdout:526: done (6 left) 2024-04-24T01:57:40.505 INFO:tasks.rados.rados.0.smithi079.stdout:527: done (5 left) 2024-04-24T01:57:40.505 INFO:tasks.rados.rados.0.smithi079.stdout:528: done (4 left) 2024-04-24T01:57:40.505 INFO:tasks.rados.rados.0.smithi079.stdout:529: done (3 left) 2024-04-24T01:57:40.505 INFO:tasks.rados.rados.0.smithi079.stdout:533: done (2 left) 2024-04-24T01:57:40.505 INFO:tasks.rados.rados.0.smithi079.stdout:534: write (excl) oid 26 current snap is 70 2024-04-24T01:57:40.505 INFO:tasks.rados.rados.0.smithi079.stdout:531: finishing write tid 1 to smithi07929762-30 2024-04-24T01:57:40.505 INFO:tasks.rados.rados.0.smithi079.stdout:534: seq_num 259 ranges {753013=645493,2056248=296765} 2024-04-24T01:57:40.511 INFO:tasks.rados.rados.0.smithi079.stdout:534: writing smithi07929762-26 from 753013 to 1398506 tid 1 2024-04-24T01:57:40.517 INFO:tasks.rados.rados.0.smithi079.stdout:534: writing smithi07929762-26 from 2056248 to 2353013 tid 2 2024-04-24T01:57:40.517 INFO:tasks.rados.rados.0.smithi079.stdout:535: write (excl) oid 19 current snap is 70 2024-04-24T01:57:40.517 INFO:tasks.rados.rados.0.smithi079.stdout:535: seq_num 260 ranges {769820=720742,2114730=255090} 2024-04-24T01:57:40.529 INFO:tasks.rados.rados.0.smithi079.stdout:535: writing smithi07929762-19 from 769820 to 1490562 tid 1 2024-04-24T01:57:40.535 INFO:tasks.rados.rados.0.smithi079.stdout:535: writing smithi07929762-19 from 2114730 to 2369820 tid 2 2024-04-24T01:57:40.535 INFO:tasks.rados.rados.0.smithi079.stdout:536: delete oid 33 current snap is 70 2024-04-24T01:57:40.535 INFO:tasks.rados.rados.0.smithi079.stdout:531: finishing write tid 2 to smithi07929762-30 2024-04-24T01:57:40.535 INFO:tasks.rados.rados.0.smithi079.stdout:531: finishing write tid 3 to smithi07929762-30 2024-04-24T01:57:40.535 INFO:tasks.rados.rados.0.smithi079.stdout:531: finishing write tid 4 to smithi07929762-30 2024-04-24T01:57:40.535 INFO:tasks.rados.rados.0.smithi079.stdout:531: oid 30 updating version 0 to 47 2024-04-24T01:57:40.535 INFO:tasks.rados.rados.0.smithi079.stdout:531: oid 30 version 47 is already newer than 47 2024-04-24T01:57:40.535 INFO:tasks.rados.rados.0.smithi079.stdout:531: oid 30 version 47 is already newer than 47 2024-04-24T01:57:40.535 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 30 v 47 (ObjNum 257 snap 70 seq_num 257) dirty exists 2024-04-24T01:57:40.535 INFO:tasks.rados.rados.0.smithi079.stdout:531: left oid 30 (ObjNum 257 snap 70 seq_num 257) 2024-04-24T01:57:40.536 INFO:tasks.rados.rados.0.smithi079.stdout:532: expect (ObjNum 162 snap 24 seq_num 162) 2024-04-24T01:57:40.708 INFO:tasks.rados.rados.0.smithi079.stdout:534: finishing write tid 1 to smithi07929762-26 2024-04-24T01:57:40.708 INFO:tasks.rados.rados.0.smithi079.stdout:534: finishing write tid 2 to smithi07929762-26 2024-04-24T01:57:40.708 INFO:tasks.rados.rados.0.smithi079.stdout:535: finishing write tid 1 to smithi07929762-19 2024-04-24T01:57:40.708 INFO:tasks.rados.rados.0.smithi079.stdout:534: finishing write tid 3 to smithi07929762-26 2024-04-24T01:57:40.708 INFO:tasks.rados.rados.0.smithi079.stdout:534: finishing write tid 4 to smithi07929762-26 2024-04-24T01:57:40.708 INFO:tasks.rados.rados.0.smithi079.stdout:534: oid 26 updating version 0 to 149 2024-04-24T01:57:40.708 INFO:tasks.rados.rados.0.smithi079.stdout:534: oid 26 updating version 149 to 151 2024-04-24T01:57:40.709 INFO:tasks.rados.rados.0.smithi079.stdout:534: oid 26 version 151 is already newer than 151 2024-04-24T01:57:40.709 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 26 v 151 (ObjNum 259 snap 70 seq_num 259) dirty exists 2024-04-24T01:57:40.709 INFO:tasks.rados.rados.0.smithi079.stdout:534: left oid 26 (ObjNum 259 snap 70 seq_num 259) 2024-04-24T01:57:40.709 INFO:tasks.rados.rados.0.smithi079.stdout:535: finishing write tid 2 to smithi07929762-19 2024-04-24T01:57:40.709 INFO:tasks.rados.rados.0.smithi079.stdout:531: done (4 left) 2024-04-24T01:57:40.709 INFO:tasks.rados.rados.0.smithi079.stdout:532: done (3 left) 2024-04-24T01:57:40.709 INFO:tasks.rados.rados.0.smithi079.stdout:534: done (2 left) 2024-04-24T01:57:40.709 INFO:tasks.rados.rados.0.smithi079.stdout:536: done (1 left) 2024-04-24T01:57:40.709 INFO:tasks.rados.rados.0.smithi079.stdout:537: read oid 10 snap -1 2024-04-24T01:57:40.709 INFO:tasks.rados.rados.0.smithi079.stdout:537: expect (ObjNum 147 snap 20 seq_num 147) 2024-04-24T01:57:40.709 INFO:tasks.rados.rados.0.smithi079.stdout:538: write oid 25 current snap is 70 2024-04-24T01:57:40.709 INFO:tasks.rados.rados.0.smithi079.stdout:538: seq_num 262 ranges {403434=471240,1315341=400756} 2024-04-24T01:57:40.712 INFO:tasks.rados.rados.0.smithi079.stdout:538: writing smithi07929762-25 from 403434 to 874674 tid 1 2024-04-24T01:57:40.717 INFO:tasks.rados.rados.0.smithi079.stdout:538: writing smithi07929762-25 from 1315341 to 1716097 tid 2 2024-04-24T01:57:40.718 INFO:tasks.rados.rados.0.smithi079.stdout:539: snap_remove snap 54 2024-04-24T01:57:41.056 INFO:tasks.rados.rados.0.smithi079.stdout:539: done (3 left) 2024-04-24T01:57:41.056 INFO:tasks.rados.rados.0.smithi079.stdout:540: delete oid 41 current snap is 70 2024-04-24T01:57:41.056 INFO:tasks.rados.rados.0.smithi079.stdout:535: finishing write tid 3 to smithi07929762-19 2024-04-24T01:57:41.057 INFO:tasks.rados.rados.0.smithi079.stdout:535: finishing write tid 4 to smithi07929762-19 2024-04-24T01:57:41.057 INFO:tasks.rados.rados.0.smithi079.stdout:535: oid 19 updating version 0 to 106 2024-04-24T01:57:41.057 INFO:tasks.rados.rados.0.smithi079.stdout:535: oid 19 version 106 is already newer than 104 2024-04-24T01:57:41.057 INFO:tasks.rados.rados.0.smithi079.stdout:535: oid 19 version 106 is already newer than 106 2024-04-24T01:57:41.057 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 19 v 106 (ObjNum 260 snap 70 seq_num 260) dirty exists 2024-04-24T01:57:41.057 INFO:tasks.rados.rados.0.smithi079.stdout:535: left oid 19 (ObjNum 260 snap 70 seq_num 260) 2024-04-24T01:57:41.057 INFO:tasks.rados.rados.0.smithi079.stdout:537: expect (ObjNum 147 snap 20 seq_num 147) 2024-04-24T01:57:41.155 INFO:tasks.rados.rados.0.smithi079.stdout:538: finishing write tid 1 to smithi07929762-25 2024-04-24T01:57:41.156 INFO:tasks.rados.rados.0.smithi079.stdout:538: finishing write tid 2 to smithi07929762-25 2024-04-24T01:57:41.156 INFO:tasks.rados.rados.0.smithi079.stdout:538: finishing write tid 3 to smithi07929762-25 2024-04-24T01:57:41.156 INFO:tasks.rados.rados.0.smithi079.stdout:538: finishing write tid 4 to smithi07929762-25 2024-04-24T01:57:41.156 INFO:tasks.rados.rados.0.smithi079.stdout:538: oid 25 updating version 0 to 155 2024-04-24T01:57:41.156 INFO:tasks.rados.rados.0.smithi079.stdout:538: oid 25 version 155 is already newer than 155 2024-04-24T01:57:41.156 INFO:tasks.rados.rados.0.smithi079.stdout:538: oid 25 version 155 is already newer than 155 2024-04-24T01:57:41.156 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 25 v 155 (ObjNum 262 snap 70 seq_num 262) dirty exists 2024-04-24T01:57:41.156 INFO:tasks.rados.rados.0.smithi079.stdout:538: left oid 25 (ObjNum 262 snap 70 seq_num 262) 2024-04-24T01:57:41.156 INFO:tasks.rados.rados.0.smithi079.stdout:535: done (3 left) 2024-04-24T01:57:41.156 INFO:tasks.rados.rados.0.smithi079.stdout:537: done (2 left) 2024-04-24T01:57:41.156 INFO:tasks.rados.rados.0.smithi079.stdout:538: done (1 left) 2024-04-24T01:57:41.156 INFO:tasks.rados.rados.0.smithi079.stdout:540: done (0 left) 2024-04-24T01:57:41.156 INFO:tasks.rados.rados.0.smithi079.stdout:541: write (excl) oid 44 current snap is 70 2024-04-24T01:57:41.157 INFO:tasks.rados.rados.0.smithi079.stdout:541: seq_num 264 ranges {437048=621738,1832305=404425} 2024-04-24T01:57:41.161 INFO:tasks.rados.rados.0.smithi079.stdout:541: writing smithi07929762-44 from 437048 to 1058786 tid 1 2024-04-24T01:57:41.170 INFO:tasks.rados.rados.0.smithi079.stdout:541: writing smithi07929762-44 from 1832305 to 2236730 tid 2 2024-04-24T01:57:41.171 INFO:tasks.rados.rados.0.smithi079.stdout:542: write oid 31 current snap is 70 2024-04-24T01:57:41.171 INFO:tasks.rados.rados.0.smithi079.stdout:542: seq_num 265 ranges {453855=696987,1890787=548083} 2024-04-24T01:57:41.173 INFO:tasks.rados.rados.0.smithi079.stdout:542: writing smithi07929762-31 from 453855 to 1150842 tid 1 2024-04-24T01:57:41.181 INFO:tasks.rados.rados.0.smithi079.stdout:542: writing smithi07929762-31 from 1890787 to 2438870 tid 2 2024-04-24T01:57:41.181 INFO:tasks.rados.rados.0.smithi079.stdout:543: write oid 21 current snap is 70 2024-04-24T01:57:41.181 INFO:tasks.rados.rados.0.smithi079.stdout:543: seq_num 266 ranges {470662=772236,1832916=408094} 2024-04-24T01:57:41.188 INFO:tasks.rados.rados.0.smithi079.stdout:543: writing smithi07929762-21 from 470662 to 1242898 tid 1 2024-04-24T01:57:41.194 INFO:tasks.rados.rados.0.smithi079.stdout:543: writing smithi07929762-21 from 1832916 to 2241010 tid 2 2024-04-24T01:57:41.194 INFO:tasks.rados.rados.0.smithi079.stdout:544: delete oid 45 current snap is 70 2024-04-24T01:57:41.274 INFO:tasks.rados.rados.0.smithi079.stdout:543: finishing write tid 1 to smithi07929762-21 2024-04-24T01:57:41.286 INFO:tasks.rados.rados.0.smithi079.stdout:542: finishing write tid 1 to smithi07929762-31 2024-04-24T01:57:41.287 INFO:tasks.rados.rados.0.smithi079.stdout:543: finishing write tid 2 to smithi07929762-21 2024-04-24T01:57:41.290 INFO:tasks.rados.rados.0.smithi079.stdout:543: finishing write tid 3 to smithi07929762-21 2024-04-24T01:57:41.293 INFO:tasks.rados.rados.0.smithi079.stdout:542: finishing write tid 2 to smithi07929762-31 2024-04-24T01:57:41.295 INFO:tasks.rados.rados.0.smithi079.stdout:543: finishing write tid 4 to smithi07929762-21 2024-04-24T01:57:41.295 INFO:tasks.rados.rados.0.smithi079.stdout:543: oid 21 updating version 0 to 133 2024-04-24T01:57:41.296 INFO:tasks.rados.rados.0.smithi079.stdout:543: oid 21 version 133 is already newer than 133 2024-04-24T01:57:41.296 INFO:tasks.rados.rados.0.smithi079.stdout:543: oid 21 version 133 is already newer than 133 2024-04-24T01:57:41.296 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 21 v 133 (ObjNum 266 snap 70 seq_num 266) dirty exists 2024-04-24T01:57:41.296 INFO:tasks.rados.rados.0.smithi079.stdout:543: left oid 21 (ObjNum 266 snap 70 seq_num 266) 2024-04-24T01:57:41.296 INFO:tasks.rados.rados.0.smithi079.stdout:541: finishing write tid 1 to smithi07929762-44 2024-04-24T01:57:41.297 INFO:tasks.rados.rados.0.smithi079.stdout:542: finishing write tid 3 to smithi07929762-31 2024-04-24T01:57:41.298 INFO:tasks.rados.rados.0.smithi079.stdout:542: finishing write tid 4 to smithi07929762-31 2024-04-24T01:57:41.298 INFO:tasks.rados.rados.0.smithi079.stdout:542: oid 31 updating version 0 to 191 2024-04-24T01:57:41.298 INFO:tasks.rados.rados.0.smithi079.stdout:542: oid 31 version 191 is already newer than 191 2024-04-24T01:57:41.298 INFO:tasks.rados.rados.0.smithi079.stdout:542: oid 31 version 191 is already newer than 191 2024-04-24T01:57:41.299 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 31 v 191 (ObjNum 265 snap 70 seq_num 265) dirty exists 2024-04-24T01:57:41.299 INFO:tasks.rados.rados.0.smithi079.stdout:542: left oid 31 (ObjNum 265 snap 70 seq_num 265) 2024-04-24T01:57:41.299 INFO:tasks.rados.rados.0.smithi079.stdout:542: done (3 left) 2024-04-24T01:57:41.299 INFO:tasks.rados.rados.0.smithi079.stdout:543: done (2 left) 2024-04-24T01:57:41.299 INFO:tasks.rados.rados.0.smithi079.stdout:544: done (1 left) 2024-04-24T01:57:41.300 INFO:tasks.rados.rados.0.smithi079.stdout:545: write oid 24 current snap is 70 2024-04-24T01:57:41.300 INFO:tasks.rados.rados.0.smithi079.stdout:545: seq_num 268 ranges {504276=639087,1666233=695410} 2024-04-24T01:57:41.308 INFO:tasks.rados.rados.0.smithi079.stdout:545: writing smithi07929762-24 from 504276 to 1143363 tid 1 2024-04-24T01:57:41.316 INFO:tasks.rados.rados.0.smithi079.stdout:545: writing smithi07929762-24 from 1666233 to 2361643 tid 2 2024-04-24T01:57:41.316 INFO:tasks.rados.rados.0.smithi079.stdout:546: read oid 22 snap 63 2024-04-24T01:57:41.316 INFO:tasks.rados.rados.0.smithi079.stdout:546: expect (ObjNum 195 snap 40 seq_num 195) 2024-04-24T01:57:41.316 INFO:tasks.rados.rados.0.smithi079.stdout:547: write oid 17 current snap is 70 2024-04-24T01:57:41.316 INFO:tasks.rados.rados.0.smithi079.stdout:541: finishing write tid 2 to smithi07929762-44 2024-04-24T01:57:41.316 INFO:tasks.rados.rados.0.smithi079.stdout:547: seq_num 269 ranges {521083=714336,1724715=555421} 2024-04-24T01:57:41.323 INFO:tasks.rados.rados.0.smithi079.stdout:547: writing smithi07929762-17 from 521083 to 1235419 tid 1 2024-04-24T01:57:41.329 INFO:tasks.rados.rados.0.smithi079.stdout:547: writing smithi07929762-17 from 1724715 to 2280136 tid 2 2024-04-24T01:57:41.329 INFO:tasks.rados.rados.0.smithi079.stdout:548: snap_remove snap 66 2024-04-24T01:57:41.828 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T01:57:41.903 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-04-24T01:57:41.903 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-04-24T01:57:41.903 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-04-24T01:57:41.904 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-04-24T01:57:41.938 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:41.936+0000 7fa0be2ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:41.942 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:41.942+0000 7fa0be2ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:42.016 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:42.011+0000 7f3b91eed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:42.020 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:42.021+0000 7f3b91eed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:42.037 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T01:57:42.037 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T01:57:42.037 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T01:57:42.037 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T01:57:42.037 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T01:57:42.037 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T01:57:42.037 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T01:57:42.037 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T01:57:42.038 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T01:57:42.038 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T01:57:42.038 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T01:57:42.038 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T01:57:42.040 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T01:57:42.040 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T01:57:42.040 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-04-24T01:57:42.059 INFO:tasks.rados.rados.0.smithi079.stdout:548: done (4 left) 2024-04-24T01:57:42.060 INFO:tasks.rados.rados.0.smithi079.stdout:549: write oid 1 current snap is 70 2024-04-24T01:57:42.060 INFO:tasks.rados.rados.0.smithi079.stdout:541: finishing write tid 3 to smithi07929762-44 2024-04-24T01:57:42.060 INFO:tasks.rados.rados.0.smithi079.stdout:549: seq_num 270 ranges {537890=789585,1783197=699079} 2024-04-24T01:57:42.067 INFO:tasks.rados.rados.0.smithi079.stdout:549: writing smithi07929762-1 from 537890 to 1327475 tid 1 2024-04-24T01:57:42.075 INFO:tasks.rados.rados.0.smithi079.stdout:549: writing smithi07929762-1 from 1783197 to 2482276 tid 2 2024-04-24T01:57:42.075 INFO:tasks.rados.rados.0.smithi079.stdout:550: write (excl) oid 39 current snap is 70 2024-04-24T01:57:42.075 INFO:tasks.rados.rados.0.smithi079.stdout:541: finishing write tid 4 to smithi07929762-44 2024-04-24T01:57:42.075 INFO:tasks.rados.rados.0.smithi079.stdout:541: oid 44 updating version 0 to 88 2024-04-24T01:57:42.075 INFO:tasks.rados.rados.0.smithi079.stdout:541: oid 44 version 88 is already newer than 88 2024-04-24T01:57:42.075 INFO:tasks.rados.rados.0.smithi079.stdout:541: oid 44 version 88 is already newer than 86 2024-04-24T01:57:42.075 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 44 v 88 (ObjNum 264 snap 70 seq_num 264) dirty exists 2024-04-24T01:57:42.075 INFO:tasks.rados.rados.0.smithi079.stdout:541: left oid 44 (ObjNum 264 snap 70 seq_num 264) 2024-04-24T01:57:42.075 INFO:tasks.rados.rados.0.smithi079.stdout:550: seq_num 271 ranges {554697=464834,1725326=559090} 2024-04-24T01:57:42.081 INFO:tasks.rados.rados.0.smithi079.stdout:550: writing smithi07929762-39 from 554697 to 1019531 tid 1 2024-04-24T01:57:42.090 INFO:tasks.rados.rados.0.smithi079.stdout:550: writing smithi07929762-39 from 1725326 to 2284416 tid 2 2024-04-24T01:57:42.090 INFO:tasks.rados.rados.0.smithi079.stdout:541: done (5 left) 2024-04-24T01:57:42.090 INFO:tasks.rados.rados.0.smithi079.stdout:551: snap_remove snap 46 2024-04-24T01:57:42.217 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:42.211+0000 7f3d9c15d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:42.224 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:42.224+0000 7f3d9c15d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:42.311 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T01:57:42.311 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T01:57:42.311 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T01:57:42.311 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T01:57:42.311 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T01:57:42.311 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T01:57:42.311 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T01:57:42.311 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T01:57:42.311 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T01:57:42.311 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T01:57:42.312 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T01:57:42.312 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T01:57:42.315 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T01:57:42.315 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T01:57:42.315 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-04-24T01:57:42.464 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:42.459+0000 7f7b33aad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:42.468 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:42.468+0000 7f7b33aad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:42.484 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T01:57:42.484 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T01:57:42.484 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T01:57:42.484 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T01:57:42.484 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T01:57:42.484 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T01:57:42.484 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T01:57:42.485 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T01:57:42.485 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T01:57:42.485 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T01:57:42.485 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T01:57:42.485 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T01:57:42.487 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T01:57:42.487 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T01:57:42.487 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-04-24T01:57:42.596 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:42.591+0000 7f971819d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:42.599 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:42.600+0000 7f971819d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:42.616 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T01:57:42.616 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T01:57:42.616 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T01:57:42.616 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T01:57:42.616 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T01:57:42.616 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T01:57:42.616 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T01:57:42.617 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T01:57:42.617 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T01:57:42.617 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T01:57:42.617 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T01:57:42.617 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T01:57:42.619 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T01:57:42.619 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T01:57:43.068 INFO:tasks.rados.rados.0.smithi079.stdout:551: done (5 left) 2024-04-24T01:57:43.068 INFO:tasks.rados.rados.0.smithi079.stdout:552: write (excl) oid 25 current snap is 70 2024-04-24T01:57:43.068 INFO:tasks.rados.rados.0.smithi079.stdout:545: finishing write tid 1 to smithi07929762-24 2024-04-24T01:57:43.068 INFO:tasks.rados.rados.0.smithi079.stdout:552: seq_num 272 ranges {571504=540083,1783808=702748} 2024-04-24T01:57:43.075 INFO:tasks.rados.rados.0.smithi079.stdout:552: writing smithi07929762-25 from 571504 to 1111587 tid 1 2024-04-24T01:57:43.084 INFO:tasks.rados.rados.0.smithi079.stdout:552: writing smithi07929762-25 from 1783808 to 2486556 tid 2 2024-04-24T01:57:43.084 INFO:tasks.rados.rados.0.smithi079.stdout:553: read oid 16 snap 58 2024-04-24T01:57:43.084 INFO:tasks.rados.rados.0.smithi079.stdout:553: expect (ObjNum 176 snap 33 seq_num 176) 2024-04-24T01:57:43.084 INFO:tasks.rados.rados.0.smithi079.stdout:554: read oid 7 snap -1 2024-04-24T01:57:43.084 INFO:tasks.rados.rados.0.smithi079.stdout:554: expect (ObjNum 167 snap 26 seq_num 167) 2024-04-24T01:57:43.084 INFO:tasks.rados.rados.0.smithi079.stdout:555: delete oid 11 current snap is 70 2024-04-24T01:57:43.084 INFO:tasks.rados.rados.0.smithi079.stdout:545: finishing write tid 2 to smithi07929762-24 2024-04-24T01:57:43.085 INFO:tasks.rados.rados.0.smithi079.stdout:545: finishing write tid 3 to smithi07929762-24 2024-04-24T01:57:43.085 INFO:tasks.rados.rados.0.smithi079.stdout:545: finishing write tid 4 to smithi07929762-24 2024-04-24T01:57:43.085 INFO:tasks.rados.rados.0.smithi079.stdout:545: oid 24 updating version 0 to 114 2024-04-24T01:57:43.085 INFO:tasks.rados.rados.0.smithi079.stdout:545: oid 24 version 114 is already newer than 114 2024-04-24T01:57:43.085 INFO:tasks.rados.rados.0.smithi079.stdout:545: oid 24 version 114 is already newer than 114 2024-04-24T01:57:43.085 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 24 v 114 (ObjNum 268 snap 70 seq_num 268) dirty exists 2024-04-24T01:57:43.085 INFO:tasks.rados.rados.0.smithi079.stdout:545: left oid 24 (ObjNum 268 snap 70 seq_num 268) 2024-04-24T01:57:43.085 INFO:tasks.rados.rados.0.smithi079.stdout:547: finishing write tid 1 to smithi07929762-17 2024-04-24T01:57:43.085 INFO:tasks.rados.rados.0.smithi079.stdout:547: finishing write tid 2 to smithi07929762-17 2024-04-24T01:57:43.085 INFO:tasks.rados.rados.0.smithi079.stdout:546: expect (ObjNum 195 snap 40 seq_num 195) 2024-04-24T01:57:43.238 INFO:tasks.rados.rados.0.smithi079.stdout:547: finishing write tid 3 to smithi07929762-17 2024-04-24T01:57:43.238 INFO:tasks.rados.rados.0.smithi079.stdout:547: finishing write tid 4 to smithi07929762-17 2024-04-24T01:57:43.238 INFO:tasks.rados.rados.0.smithi079.stdout:547: oid 17 updating version 0 to 138 2024-04-24T01:57:43.238 INFO:tasks.rados.rados.0.smithi079.stdout:547: oid 17 version 138 is already newer than 138 2024-04-24T01:57:43.239 INFO:tasks.rados.rados.0.smithi079.stdout:547: oid 17 version 138 is already newer than 138 2024-04-24T01:57:43.239 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 17 v 138 (ObjNum 269 snap 70 seq_num 269) dirty exists 2024-04-24T01:57:43.239 INFO:tasks.rados.rados.0.smithi079.stdout:547: left oid 17 (ObjNum 269 snap 70 seq_num 269) 2024-04-24T01:57:43.239 INFO:tasks.rados.rados.0.smithi079.stdout:549: finishing write tid 1 to smithi07929762-1 2024-04-24T01:57:43.239 INFO:tasks.rados.rados.0.smithi079.stdout:549: finishing write tid 2 to smithi07929762-1 2024-04-24T01:57:43.239 INFO:tasks.rados.rados.0.smithi079.stdout:549: finishing write tid 3 to smithi07929762-1 2024-04-24T01:57:43.240 INFO:tasks.rados.rados.0.smithi079.stdout:550: finishing write tid 1 to smithi07929762-39 2024-04-24T01:57:43.240 INFO:tasks.rados.rados.0.smithi079.stdout:549: finishing write tid 4 to smithi07929762-1 2024-04-24T01:57:43.240 INFO:tasks.rados.rados.0.smithi079.stdout:549: oid 1 updating version 0 to 197 2024-04-24T01:57:43.240 INFO:tasks.rados.rados.0.smithi079.stdout:549: oid 1 version 197 is already newer than 197 2024-04-24T01:57:43.240 INFO:tasks.rados.rados.0.smithi079.stdout:549: oid 1 version 197 is already newer than 197 2024-04-24T01:57:43.240 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 1 v 197 (ObjNum 270 snap 70 seq_num 270) dirty exists 2024-04-24T01:57:43.241 INFO:tasks.rados.rados.0.smithi079.stdout:549: left oid 1 (ObjNum 270 snap 70 seq_num 270) 2024-04-24T01:57:43.241 INFO:tasks.rados.rados.0.smithi079.stdout:550: finishing write tid 2 to smithi07929762-39 2024-04-24T01:57:43.241 INFO:tasks.rados.rados.0.smithi079.stdout:550: finishing write tid 3 to smithi07929762-39 2024-04-24T01:57:43.241 INFO:tasks.rados.rados.0.smithi079.stdout:550: finishing write tid 4 to smithi07929762-39 2024-04-24T01:57:43.241 INFO:tasks.rados.rados.0.smithi079.stdout:550: oid 39 updating version 0 to 92 2024-04-24T01:57:43.241 INFO:tasks.rados.rados.0.smithi079.stdout:550: oid 39 updating version 92 to 94 2024-04-24T01:57:43.242 INFO:tasks.rados.rados.0.smithi079.stdout:550: oid 39 version 94 is already newer than 94 2024-04-24T01:57:43.242 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 39 v 94 (ObjNum 271 snap 70 seq_num 271) dirty exists 2024-04-24T01:57:43.242 INFO:tasks.rados.rados.0.smithi079.stdout:550: left oid 39 (ObjNum 271 snap 70 seq_num 271) 2024-04-24T01:57:43.242 INFO:tasks.rados.rados.0.smithi079.stdout:554: expect (ObjNum 167 snap 26 seq_num 167) 2024-04-24T01:57:43.429 INFO:tasks.rados.rados.0.smithi079.stdout:553: expect (ObjNum 176 snap 33 seq_num 176) 2024-04-24T01:57:43.539 INFO:tasks.rados.rados.0.smithi079.stdout:552: finishing write tid 1 to smithi07929762-25 2024-04-24T01:57:43.539 INFO:tasks.rados.rados.0.smithi079.stdout:552: finishing write tid 2 to smithi07929762-25 2024-04-24T01:57:43.539 INFO:tasks.rados.rados.0.smithi079.stdout:552: finishing write tid 3 to smithi07929762-25 2024-04-24T01:57:43.540 INFO:tasks.rados.rados.0.smithi079.stdout:552: finishing write tid 4 to smithi07929762-25 2024-04-24T01:57:43.540 INFO:tasks.rados.rados.0.smithi079.stdout:552: oid 25 updating version 0 to 172 2024-04-24T01:57:43.540 INFO:tasks.rados.rados.0.smithi079.stdout:552: oid 25 version 172 is already newer than 170 2024-04-24T01:57:43.540 INFO:tasks.rados.rados.0.smithi079.stdout:552: oid 25 version 172 is already newer than 172 2024-04-24T01:57:43.540 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 25 v 172 (ObjNum 272 snap 70 seq_num 272) dirty exists 2024-04-24T01:57:43.540 INFO:tasks.rados.rados.0.smithi079.stdout:552: left oid 25 (ObjNum 272 snap 70 seq_num 272) 2024-04-24T01:57:43.540 INFO:tasks.rados.rados.0.smithi079.stdout:545: done (8 left) 2024-04-24T01:57:43.540 INFO:tasks.rados.rados.0.smithi079.stdout:546: done (7 left) 2024-04-24T01:57:43.540 INFO:tasks.rados.rados.0.smithi079.stdout:547: done (6 left) 2024-04-24T01:57:43.540 INFO:tasks.rados.rados.0.smithi079.stdout:549: done (5 left) 2024-04-24T01:57:43.540 INFO:tasks.rados.rados.0.smithi079.stdout:550: done (4 left) 2024-04-24T01:57:43.540 INFO:tasks.rados.rados.0.smithi079.stdout:552: done (3 left) 2024-04-24T01:57:43.540 INFO:tasks.rados.rados.0.smithi079.stdout:553: done (2 left) 2024-04-24T01:57:43.540 INFO:tasks.rados.rados.0.smithi079.stdout:554: done (1 left) 2024-04-24T01:57:43.540 INFO:tasks.rados.rados.0.smithi079.stdout:555: done (0 left) 2024-04-24T01:57:43.541 INFO:tasks.rados.rados.0.smithi079.stdout:556: snap_remove snap 58 2024-04-24T01:57:44.039 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T01:57:44.074 INFO:tasks.rados.rados.0.smithi079.stdout:556: done (0 left) 2024-04-24T01:57:44.074 INFO:tasks.rados.rados.0.smithi079.stdout:557: read oid 22 snap -1 2024-04-24T01:57:44.074 INFO:tasks.rados.rados.0.smithi079.stdout:557: expect (ObjNum 253 snap 70 seq_num 253) 2024-04-24T01:57:44.074 INFO:tasks.rados.rados.0.smithi079.stdout:558: write oid 7 current snap is 70 2024-04-24T01:57:44.074 INFO:tasks.rados.rados.0.smithi079.stdout:558: seq_num 274 ranges {605118=406934,1617125=706417} 2024-04-24T01:57:44.081 INFO:tasks.rados.rados.0.smithi079.stdout:558: writing smithi07929762-7 from 605118 to 1012052 tid 1 2024-04-24T01:57:44.093 INFO:tasks.rados.rados.0.smithi079.stdout:558: writing smithi07929762-7 from 1617125 to 2323542 tid 2 2024-04-24T01:57:44.093 INFO:tasks.rados.rados.0.smithi079.stdout:559: write oid 24 current snap is 70 2024-04-24T01:57:44.093 INFO:tasks.rados.rados.0.smithi079.stdout:559: seq_num 275 ranges {621925=482183,1559254=566428} 2024-04-24T01:57:44.101 INFO:tasks.rados.rados.0.smithi079.stdout:559: writing smithi07929762-24 from 621925 to 1104108 tid 1 2024-04-24T01:57:44.110 INFO:tasks.rados.rados.0.smithi079.stdout:559: writing smithi07929762-24 from 1559254 to 2125682 tid 2 2024-04-24T01:57:44.110 INFO:tasks.rados.rados.0.smithi079.stdout:560: read oid 20 snap 69 2024-04-24T01:57:44.110 INFO:tasks.rados.rados.0.smithi079.stdout:560: expect deleted 2024-04-24T01:57:44.110 INFO:tasks.rados.rados.0.smithi079.stdout:561: snap_remove snap 50 2024-04-24T01:57:44.184 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:44.180+0000 7fe78d4ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:44.189 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:44.190+0000 7fe78d4ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:45.077 INFO:tasks.rados.rados.0.smithi079.stdout:561: done (4 left) 2024-04-24T01:57:45.077 INFO:tasks.rados.rados.0.smithi079.stdout:562: snap_create 2024-04-24T01:57:45.077 INFO:tasks.rados.rados.0.smithi079.stdout:557: expect (ObjNum 253 snap 70 seq_num 253) 2024-04-24T01:57:45.191 INFO:tasks.rados.rados.0.smithi079.stdout:559: finishing write tid 1 to smithi07929762-24 2024-04-24T01:57:45.191 INFO:tasks.rados.rados.0.smithi079.stdout:558: finishing write tid 1 to smithi07929762-7 2024-04-24T01:57:45.191 INFO:tasks.rados.rados.0.smithi079.stdout:559: finishing write tid 2 to smithi07929762-24 2024-04-24T01:57:45.191 INFO:tasks.rados.rados.0.smithi079.stdout:559: finishing write tid 3 to smithi07929762-24 2024-04-24T01:57:45.191 INFO:tasks.rados.rados.0.smithi079.stdout:559: finishing write tid 4 to smithi07929762-24 2024-04-24T01:57:45.191 INFO:tasks.rados.rados.0.smithi079.stdout:559: oid 24 updating version 0 to 129 2024-04-24T01:57:45.191 INFO:tasks.rados.rados.0.smithi079.stdout:559: oid 24 version 129 is already newer than 129 2024-04-24T01:57:45.191 INFO:tasks.rados.rados.0.smithi079.stdout:559: oid 24 version 129 is already newer than 129 2024-04-24T01:57:45.191 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 24 v 129 (ObjNum 275 snap 70 seq_num 275) dirty exists 2024-04-24T01:57:45.191 INFO:tasks.rados.rados.0.smithi079.stdout:559: left oid 24 (ObjNum 275 snap 70 seq_num 275) 2024-04-24T01:57:45.191 INFO:tasks.rados.rados.0.smithi079.stdout:558: finishing write tid 2 to smithi07929762-7 2024-04-24T01:57:45.191 INFO:tasks.rados.rados.0.smithi079.stdout:558: finishing write tid 3 to smithi07929762-7 2024-04-24T01:57:45.191 INFO:tasks.rados.rados.0.smithi079.stdout:558: finishing write tid 4 to smithi07929762-7 2024-04-24T01:57:45.192 INFO:tasks.rados.rados.0.smithi079.stdout:558: oid 7 updating version 0 to 104 2024-04-24T01:57:45.192 INFO:tasks.rados.rados.0.smithi079.stdout:558: oid 7 version 104 is already newer than 104 2024-04-24T01:57:45.192 INFO:tasks.rados.rados.0.smithi079.stdout:558: oid 7 version 104 is already newer than 104 2024-04-24T01:57:45.192 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 7 v 104 (ObjNum 274 snap 70 seq_num 274) dirty exists 2024-04-24T01:57:45.192 INFO:tasks.rados.rados.0.smithi079.stdout:558: left oid 7 (ObjNum 274 snap 70 seq_num 274) 2024-04-24T01:57:46.033 INFO:tasks.rados.rados.0.smithi079.stdout:557: done (4 left) 2024-04-24T01:57:46.034 INFO:tasks.rados.rados.0.smithi079.stdout:558: done (3 left) 2024-04-24T01:57:46.034 INFO:tasks.rados.rados.0.smithi079.stdout:559: done (2 left) 2024-04-24T01:57:46.034 INFO:tasks.rados.rados.0.smithi079.stdout:560: done (1 left) 2024-04-24T01:57:46.034 INFO:tasks.rados.rados.0.smithi079.stdout:562: done (0 left) 2024-04-24T01:57:46.034 INFO:tasks.rados.rados.0.smithi079.stdout:563: write oid 6 current snap is 71 2024-04-24T01:57:46.034 INFO:tasks.rados.rados.0.smithi079.stdout:563: seq_num 276 ranges {638732=557432,1617736=710086} 2024-04-24T01:57:46.040 INFO:tasks.rados.rados.0.smithi079.stdout:563: writing smithi07929762-6 from 638732 to 1196164 tid 1 2024-04-24T01:57:46.046 INFO:tasks.rados.rados.0.smithi079.stdout:563: writing smithi07929762-6 from 1617736 to 2327822 tid 2 2024-04-24T01:57:46.046 INFO:tasks.rados.rados.0.smithi079.stdout:564: snap_create 2024-04-24T01:57:46.171 INFO:tasks.rados.rados.0.smithi079.stdout:563: finishing write tid 1 to smithi07929762-6 2024-04-24T01:57:46.178 INFO:tasks.rados.rados.0.smithi079.stdout:563: finishing write tid 2 to smithi07929762-6 2024-04-24T01:57:46.184 INFO:tasks.rados.rados.0.smithi079.stdout:563: finishing write tid 3 to smithi07929762-6 2024-04-24T01:57:46.185 INFO:tasks.rados.rados.0.smithi079.stdout:563: finishing write tid 4 to smithi07929762-6 2024-04-24T01:57:46.185 INFO:tasks.rados.rados.0.smithi079.stdout:563: oid 6 updating version 0 to 152 2024-04-24T01:57:46.186 INFO:tasks.rados.rados.0.smithi079.stdout:563: oid 6 version 152 is already newer than 152 2024-04-24T01:57:46.186 INFO:tasks.rados.rados.0.smithi079.stdout:563: oid 6 version 152 is already newer than 152 2024-04-24T01:57:46.186 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 6 v 152 (ObjNum 276 snap 71 seq_num 276) dirty exists 2024-04-24T01:57:46.186 INFO:tasks.rados.rados.0.smithi079.stdout:563: left oid 6 (ObjNum 276 snap 71 seq_num 276) 2024-04-24T01:57:46.320 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T01:57:46.446 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:46.445+0000 7f6ca7aad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:46.450 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:46.450+0000 7f6ca7aad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:47.035 INFO:tasks.rados.rados.0.smithi079.stdout:563: done (1 left) 2024-04-24T01:57:47.035 INFO:tasks.rados.rados.0.smithi079.stdout:564: done (0 left) 2024-04-24T01:57:47.035 INFO:tasks.rados.rados.0.smithi079.stdout:565: write oid 22 current snap is 72 2024-04-24T01:57:47.035 INFO:tasks.rados.rados.0.smithi079.stdout:565: seq_num 277 ranges {655539=632681,2076218=570097} 2024-04-24T01:57:47.042 INFO:tasks.rados.rados.0.smithi079.stdout:565: writing smithi07929762-22 from 655539 to 1288220 tid 1 2024-04-24T01:57:47.049 INFO:tasks.rados.rados.0.smithi079.stdout:565: writing smithi07929762-22 from 2076218 to 2646315 tid 2 2024-04-24T01:57:47.049 INFO:tasks.rados.rados.0.smithi079.stdout:566: snap_remove snap 69 2024-04-24T01:57:47.620 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-04-24T01:57:47.621 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-04-24T01:57:47.621 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T01:57:47.733 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:47.730+0000 7fb7a506d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:47.737 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:47.737+0000 7fb7a506d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:47.923 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:57:47.924 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-24T01:57:47.936 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":221,"stamp":"2024-04-24T01:57:47.033856+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":95617674,"num_objects":94,"num_object_clones":49,"num_object_copies":282,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":1203,"num_read_kb":756990,"num_write":1113,"num_write_kb":270659,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1748,"ondisk_log_size":1748,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":504300,"kb_used_data":379200,"kb_used_omap":0,"kb_used_meta":123904,"kb_avail":376983060,"statfs":{"total":386547056640,"available":386030653440,"internally_reserved":0,"allocated":388300800,"data_stored":436917350,"data_compressed":89889,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":126877696},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-10436883,"num_objects":9,"num_object_clones":9,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":55,"num_read_kb":38139,"num_write":53,"num_write_kb":13733,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.967533"},"pg_stats":[{"pgid":"3.f","version":"149'81","reported_seq":491,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-04-24T01:57:43.102761+0000","last_change":"2024-04-24T01:57:43.216787+0000","last_active":"2024-04-24T01:57:43.102761+0000","last_peered":"2024-04-24T01:57:43.102761+0000","last_clean":"2024-04-24T01:57:43.102761+0000","last_became_active":"2024-04-24T01:55:30.386726+0000","last_became_peered":"2024-04-24T01:55:30.386726+0000","last_unstale":"2024-04-24T01:57:43.102761+0000","last_undegraded":"2024-04-24T01:57:43.102761+0000","last_fullsized":"2024-04-24T01:57:43.102761+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":5,"snaptrim_duration":0,"stat_sum":{"num_bytes":3512663,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":61,"num_read_kb":56852,"num_write":38,"num_write_kb":11452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"150'143","reported_seq":556,"reported_epoch":150,"state":"active+clean+snaptrim","last_fresh":"2024-04-24T01:57:44.123487+0000","last_change":"2024-04-24T01:57:44.099846+0000","last_active":"2024-04-24T01:57:44.123487+0000","last_peered":"2024-04-24T01:57:44.123487+0000","last_clean":"2024-04-24T01:57:44.123487+0000","last_became_active":"2024-04-24T01:55:30.321866+0000","last_became_peered":"2024-04-24T01:55:30.321866+0000","last_unstale":"2024-04-24T01:57:44.123487+0000","last_undegraded":"2024-04-24T01:57:44.123487+0000","last_fullsized":"2024-04-24T01:57:44.123487+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":143,"log_dups_size":0,"ondisk_log_size":143,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":8,"snaptrim_duration":0,"stat_sum":{"num_bytes":8016972,"num_objects":8,"num_object_clones":4,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":95,"num_read_kb":57036,"num_write":81,"num_write_kb":22457,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"149'129","reported_seq":529,"reported_epoch":149,"state":"active+clean+snaptrim","last_fresh":"2024-04-24T01:57:43.109507+0000","last_change":"2024-04-24T01:57:43.092792+0000","last_active":"2024-04-24T01:57:43.109507+0000","last_peered":"2024-04-24T01:57:43.109507+0000","last_clean":"2024-04-24T01:57:43.109507+0000","last_became_active":"2024-04-24T01:55:30.320186+0000","last_became_peered":"2024-04-24T01:55:30.320186+0000","last_unstale":"2024-04-24T01:57:43.109507+0000","last_undegraded":"2024-04-24T01:57:43.109507+0000","last_fullsized":"2024-04-24T01:57:43.109507+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":11,"snaptrim_duration":0,"stat_sum":{"num_bytes":7512729,"num_objects":6,"num_object_clones":3,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":50,"num_read_kb":16608,"num_write":84,"num_write_kb":23135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"149'116","reported_seq":526,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-04-24T01:57:43.191525+0000","last_change":"2024-04-24T01:57:43.191525+0000","last_active":"2024-04-24T01:57:43.191525+0000","last_peered":"2024-04-24T01:57:43.191525+0000","last_clean":"2024-04-24T01:57:43.191525+0000","last_became_active":"2024-04-24T01:55:30.288081+0000","last_became_peered":"2024-04-24T01:55:30.288081+0000","last_unstale":"2024-04-24T01:57:43.191525+0000","last_undegraded":"2024-04-24T01:57:43.191525+0000","last_fullsized":"2024-04-24T01:57:43.191525+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":10,"snaptrim_duration":0,"stat_sum":{"num_bytes":8033779,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":87,"num_read_kb":65894,"num_write":55,"num_write_kb":16107,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"149'52","reported_seq":472,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-04-24T01:57:44.107041+0000","last_change":"2024-04-24T01:57:44.110628+0000","last_active":"2024-04-24T01:57:44.107041+0000","last_peered":"2024-04-24T01:57:44.107041+0000","last_clean":"2024-04-24T01:57:44.107041+0000","last_became_active":"2024-04-24T01:55:30.333815+0000","last_became_peered":"2024-04-24T01:55:30.333815+0000","last_unstale":"2024-04-24T01:57:44.107041+0000","last_undegraded":"2024-04-24T01:57:44.107041+0000","last_fullsized":"2024-04-24T01:57:44.107041+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0,"stat_sum":{"num_bytes":3798382,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":32,"num_read_kb":17556,"num_write":36,"num_write_kb":10196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"149'209","reported_seq":631,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-04-24T01:57:43.280137+0000","last_change":"2024-04-24T01:57:43.271772+0000","last_active":"2024-04-24T01:57:43.280137+0000","last_peered":"2024-04-24T01:57:43.280137+0000","last_clean":"2024-04-24T01:57:43.280137+0000","last_became_active":"2024-04-24T01:55:30.315075+0000","last_became_peered":"2024-04-24T01:55:30.315075+0000","last_unstale":"2024-04-24T01:57:43.280137+0000","last_undegraded":"2024-04-24T01:57:43.280137+0000","last_fullsized":"2024-04-24T01:57:43.280137+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":209,"log_dups_size":0,"ondisk_log_size":209,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":16,"snaptrim_duration":0,"stat_sum":{"num_bytes":14907941,"num_objects":14,"num_object_clones":8,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":149,"num_read_kb":91286,"num_write":133,"num_write_kb":36375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"112'37","reported_seq":448,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-04-24T01:57:43.086933+0000","last_change":"2024-04-24T01:57:43.097580+0000","last_active":"2024-04-24T01:57:43.086933+0000","last_peered":"2024-04-24T01:57:43.086933+0000","last_clean":"2024-04-24T01:57:43.086933+0000","last_became_active":"2024-04-24T01:55:30.350300+0000","last_became_peered":"2024-04-24T01:55:30.350300+0000","last_unstale":"2024-04-24T01:57:43.086933+0000","last_undegraded":"2024-04-24T01:57:43.086933+0000","last_fullsized":"2024-04-24T01:57:43.086933+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0,"stat_sum":{"num_bytes":3462242,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":23,"num_read_kb":13068,"num_write":31,"num_write_kb":7547,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"97'7","reported_seq":416,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-04-24T01:57:43.113370+0000","last_change":"2024-04-24T01:57:43.122482+0000","last_active":"2024-04-24T01:57:43.113370+0000","last_peered":"2024-04-24T01:57:43.113370+0000","last_clean":"2024-04-24T01:57:43.113370+0000","last_became_active":"2024-04-24T01:55:30.376813+0000","last_became_peered":"2024-04-24T01:55:30.376813+0000","last_unstale":"2024-04-24T01:57:43.113370+0000","last_undegraded":"2024-04-24T01:57:43.113370+0000","last_fullsized":"2024-04-24T01:57:43.113370+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'97","reported_seq":498,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-04-24T01:57:44.090569+0000","last_change":"2024-04-24T01:55:16.196470+0000","last_active":"2024-04-24T01:57:44.090569+0000","last_peered":"2024-04-24T01:57:44.090569+0000","last_clean":"2024-04-24T01:57:44.090569+0000","last_became_active":"2024-04-24T01:55:16.190464+0000","last_became_peered":"2024-04-24T01:55:16.190464+0000","last_unstale":"2024-04-24T01:57:44.090569+0000","last_undegraded":"2024-04-24T01:57:44.090569+0000","last_fullsized":"2024-04-24T01:57:44.090569+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:15.127115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:15.127115+0000","last_clean_scrub_stamp":"2024-04-24T01:55:15.127115+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":360,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-04-24T01:57:43.113042+0000","last_change":"2024-04-24T01:55:18.236148+0000","last_active":"2024-04-24T01:57:43.113042+0000","last_peered":"2024-04-24T01:57:43.113042+0000","last_clean":"2024-04-24T01:57:43.113042+0000","last_became_active":"2024-04-24T01:55:18.230154+0000","last_became_peered":"2024-04-24T01:55:18.230154+0000","last_unstale":"2024-04-24T01:57:43.113042+0000","last_undegraded":"2024-04-24T01:57:43.113042+0000","last_fullsized":"2024-04-24T01:57:43.113042+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"149'167","reported_seq":597,"reported_epoch":149,"state":"active+clean+snaptrim","last_fresh":"2024-04-24T01:57:43.130873+0000","last_change":"2024-04-24T01:57:43.110940+0000","last_active":"2024-04-24T01:57:43.130873+0000","last_peered":"2024-04-24T01:57:43.130873+0000","last_clean":"2024-04-24T01:57:43.130873+0000","last_became_active":"2024-04-24T01:55:30.316222+0000","last_became_peered":"2024-04-24T01:55:30.316222+0000","last_unstale":"2024-04-24T01:57:43.130873+0000","last_undegraded":"2024-04-24T01:57:43.130873+0000","last_fullsized":"2024-04-24T01:57:43.130873+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":167,"log_dups_size":0,"ondisk_log_size":167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":11,"snaptrim_duration":0,"stat_sum":{"num_bytes":8000198,"num_objects":10,"num_object_clones":6,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":126,"num_read_kb":84804,"num_write":107,"num_write_kb":29713,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":356,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-04-24T01:57:43.102708+0000","last_change":"2024-04-24T01:55:18.265905+0000","last_active":"2024-04-24T01:57:43.102708+0000","last_peered":"2024-04-24T01:57:43.102708+0000","last_clean":"2024-04-24T01:57:43.102708+0000","last_became_active":"2024-04-24T01:55:18.257020+0000","last_became_peered":"2024-04-24T01:55:18.257020+0000","last_unstale":"2024-04-24T01:57:43.102708+0000","last_undegraded":"2024-04-24T01:57:43.102708+0000","last_fullsized":"2024-04-24T01:57:43.102708+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"150'124","reported_seq":540,"reported_epoch":150,"state":"active+clean+snaptrim","last_fresh":"2024-04-24T01:57:44.115281+0000","last_change":"2024-04-24T01:57:44.086301+0000","last_active":"2024-04-24T01:57:44.115281+0000","last_peered":"2024-04-24T01:57:44.115281+0000","last_clean":"2024-04-24T01:57:44.115281+0000","last_became_active":"2024-04-24T01:55:30.344129+0000","last_became_peered":"2024-04-24T01:55:30.344129+0000","last_unstale":"2024-04-24T01:57:44.115281+0000","last_undegraded":"2024-04-24T01:57:44.115281+0000","last_fullsized":"2024-04-24T01:57:44.115281+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":124,"log_dups_size":0,"ondisk_log_size":124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":5,"snaptrim_duration":0,"stat_sum":{"num_bytes":8033812,"num_objects":7,"num_object_clones":4,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":75,"num_read_kb":47563,"num_write":68,"num_write_kb":20022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":360,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-04-24T01:57:43.088566+0000","last_change":"2024-04-24T01:55:18.215251+0000","last_active":"2024-04-24T01:57:43.088566+0000","last_peered":"2024-04-24T01:57:43.088566+0000","last_clean":"2024-04-24T01:57:43.088566+0000","last_became_active":"2024-04-24T01:55:18.206605+0000","last_became_peered":"2024-04-24T01:55:18.206605+0000","last_unstale":"2024-04-24T01:57:43.088566+0000","last_undegraded":"2024-04-24T01:57:43.088566+0000","last_fullsized":"2024-04-24T01:57:43.088566+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"150'78","reported_seq":496,"reported_epoch":150,"state":"active+clean+snaptrim","last_fresh":"2024-04-24T01:57:44.143536+0000","last_change":"2024-04-24T01:57:44.098489+0000","last_active":"2024-04-24T01:57:44.143536+0000","last_peered":"2024-04-24T01:57:44.143536+0000","last_clean":"2024-04-24T01:57:44.143536+0000","last_became_active":"2024-04-24T01:55:30.401366+0000","last_became_peered":"2024-04-24T01:55:30.401366+0000","last_unstale":"2024-04-24T01:57:44.143536+0000","last_undegraded":"2024-04-24T01:57:44.143536+0000","last_fullsized":"2024-04-24T01:57:44.143536+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":3,"snaptrim_duration":0,"stat_sum":{"num_bytes":2252171,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":60,"num_read_kb":47113,"num_write":33,"num_write_kb":8517,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":356,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-04-24T01:57:43.093705+0000","last_change":"2024-04-24T01:55:18.230893+0000","last_active":"2024-04-24T01:57:43.093705+0000","last_peered":"2024-04-24T01:57:43.093705+0000","last_clean":"2024-04-24T01:57:43.093705+0000","last_became_active":"2024-04-24T01:55:18.225188+0000","last_became_peered":"2024-04-24T01:55:18.225188+0000","last_unstale":"2024-04-24T01:57:43.093705+0000","last_undegraded":"2024-04-24T01:57:43.093705+0000","last_fullsized":"2024-04-24T01:57:43.093705+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"150'100","reported_seq":518,"reported_epoch":150,"state":"active+clean+snaptrim","last_fresh":"2024-04-24T01:57:44.142005+0000","last_change":"2024-04-24T01:57:44.108448+0000","last_active":"2024-04-24T01:57:44.142005+0000","last_peered":"2024-04-24T01:57:44.142005+0000","last_clean":"2024-04-24T01:57:44.142005+0000","last_became_active":"2024-04-24T01:55:30.343318+0000","last_became_peered":"2024-04-24T01:55:30.343318+0000","last_unstale":"2024-04-24T01:57:44.142005+0000","last_undegraded":"2024-04-24T01:57:44.142005+0000","last_fullsized":"2024-04-24T01:57:44.142005+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":7,"snaptrim_duration":0,"stat_sum":{"num_bytes":7798514,"num_objects":7,"num_object_clones":4,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":79,"num_read_kb":72671,"num_write":64,"num_write_kb":18835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"148'165","reported_seq":574,"reported_epoch":149,"state":"active+clean+snaptrim","last_fresh":"2024-04-24T01:57:43.116265+0000","last_change":"2024-04-24T01:57:43.115554+0000","last_active":"2024-04-24T01:57:43.116265+0000","last_peered":"2024-04-24T01:57:43.116265+0000","last_clean":"2024-04-24T01:57:43.116265+0000","last_became_active":"2024-04-24T01:55:30.345308+0000","last_became_peered":"2024-04-24T01:55:30.345308+0000","last_unstale":"2024-04-24T01:57:43.116265+0000","last_undegraded":"2024-04-24T01:57:43.116265+0000","last_fullsized":"2024-04-24T01:57:43.116265+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":165,"log_dups_size":0,"ondisk_log_size":165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":10,"snaptrim_duration":0,"stat_sum":{"num_bytes":5058940,"num_objects":10,"num_object_clones":6,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":103,"num_read_kb":66857,"num_write":86,"num_write_kb":21276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":355,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-04-24T01:57:44.115208+0000","last_change":"2024-04-24T01:55:18.224238+0000","last_active":"2024-04-24T01:57:44.115208+0000","last_peered":"2024-04-24T01:57:44.115208+0000","last_clean":"2024-04-24T01:57:44.115208+0000","last_became_active":"2024-04-24T01:55:18.219131+0000","last_became_peered":"2024-04-24T01:55:18.219131+0000","last_unstale":"2024-04-24T01:57:44.115208+0000","last_undegraded":"2024-04-24T01:57:44.115208+0000","last_fullsized":"2024-04-24T01:57:44.115208+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"128'44","reported_seq":450,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-04-24T01:57:44.121613+0000","last_change":"2024-04-24T01:57:44.127971+0000","last_active":"2024-04-24T01:57:44.121613+0000","last_peered":"2024-04-24T01:57:44.121613+0000","last_clean":"2024-04-24T01:57:44.121613+0000","last_became_active":"2024-04-24T01:55:30.319958+0000","last_became_peered":"2024-04-24T01:55:30.319958+0000","last_unstale":"2024-04-24T01:57:44.121613+0000","last_undegraded":"2024-04-24T01:57:44.121613+0000","last_fullsized":"2024-04-24T01:57:44.121613+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":22,"num_read_kb":19606,"num_write":28,"num_write_kb":7720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":365,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-04-24T01:57:44.102104+0000","last_change":"2024-04-24T01:55:18.235763+0000","last_active":"2024-04-24T01:57:44.102104+0000","last_peered":"2024-04-24T01:57:44.102104+0000","last_clean":"2024-04-24T01:57:44.102104+0000","last_became_active":"2024-04-24T01:55:18.229175+0000","last_became_peered":"2024-04-24T01:55:18.229175+0000","last_unstale":"2024-04-24T01:57:44.102104+0000","last_undegraded":"2024-04-24T01:57:44.102104+0000","last_fullsized":"2024-04-24T01:57:44.102104+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"150'55","reported_seq":475,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-04-24T01:57:44.103605+0000","last_change":"2024-04-24T01:57:44.154026+0000","last_active":"2024-04-24T01:57:44.103605+0000","last_peered":"2024-04-24T01:57:44.103605+0000","last_clean":"2024-04-24T01:57:44.103605+0000","last_became_active":"2024-04-24T01:55:30.372690+0000","last_became_peered":"2024-04-24T01:55:30.372690+0000","last_unstale":"2024-04-24T01:57:44.103605+0000","last_undegraded":"2024-04-24T01:57:44.103605+0000","last_fullsized":"2024-04-24T01:57:44.103605+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":3,"snaptrim_duration":0,"stat_sum":{"num_bytes":6067360,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":38,"num_read_kb":24089,"num_write":34,"num_write_kb":9655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":355,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-04-24T01:57:44.116652+0000","last_change":"2024-04-24T01:55:18.278351+0000","last_active":"2024-04-24T01:57:44.116652+0000","last_peered":"2024-04-24T01:57:44.116652+0000","last_clean":"2024-04-24T01:57:44.116652+0000","last_became_active":"2024-04-24T01:55:18.273668+0000","last_became_peered":"2024-04-24T01:55:18.273668+0000","last_unstale":"2024-04-24T01:57:44.116652+0000","last_undegraded":"2024-04-24T01:57:44.116652+0000","last_fullsized":"2024-04-24T01:57:44.116652+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"149'144","reported_seq":561,"reported_epoch":150,"state":"active+clean+snaptrim","last_fresh":"2024-04-24T01:57:44.123875+0000","last_change":"2024-04-24T01:57:44.123452+0000","last_active":"2024-04-24T01:57:44.123875+0000","last_peered":"2024-04-24T01:57:44.123875+0000","last_clean":"2024-04-24T01:57:44.123875+0000","last_became_active":"2024-04-24T01:55:30.363123+0000","last_became_peered":"2024-04-24T01:55:30.363123+0000","last_unstale":"2024-04-24T01:57:44.123875+0000","last_undegraded":"2024-04-24T01:57:44.123875+0000","last_fullsized":"2024-04-24T01:57:44.123875+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":144,"log_dups_size":0,"ondisk_log_size":144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":11,"snaptrim_duration":0,"stat_sum":{"num_bytes":8571603,"num_objects":7,"num_object_clones":3,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":98,"num_read_kb":75895,"num_write":93,"num_write_kb":24910,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":360,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-04-24T01:57:43.109700+0000","last_change":"2024-04-24T01:55:18.241117+0000","last_active":"2024-04-24T01:57:43.109700+0000","last_peered":"2024-04-24T01:57:43.109700+0000","last_clean":"2024-04-24T01:57:43.109700+0000","last_became_active":"2024-04-24T01:55:18.234313+0000","last_became_peered":"2024-04-24T01:55:18.234313+0000","last_unstale":"2024-04-24T01:57:43.109700+0000","last_undegraded":"2024-04-24T01:57:43.109700+0000","last_fullsized":"2024-04-24T01:57:43.109700+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":95027306,"num_objects":92,"num_object_clones":49,"num_object_copies":276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":1100,"num_read_kb":756899,"num_write":976,"num_write_kb":268765,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":380432384,"data_stored":425494968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1651,"ondisk_log_size":1651,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":1}],"osd_stats":[{"osd":3,"up_from":10,"seq":42949673109,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":134600,"kb_used_data":103616,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94237240,"statfs":{"total":96636764160,"available":96498933760,"internally_reserved":0,"allocated":106102784,"data_stored":119412519,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673109,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":109468,"kb_used_data":78484,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94262372,"statfs":{"total":96636764160,"available":96524668928,"internally_reserved":0,"allocated":80367616,"data_stored":90705468,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705814,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":119256,"kb_used_data":87976,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94252584,"statfs":{"total":96636764160,"available":96514646016,"internally_reserved":0,"allocated":90087424,"data_stored":99013834,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705813,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":140976,"kb_used_data":109124,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94230864,"statfs":{"total":96636764160,"available":96492404736,"internally_reserved":0,"allocated":111742976,"data_stored":127785529,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":89509888,"data_stored":98140903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":106078208,"data_stored":119217517,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":79081472,"data_stored":89591399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":105762816,"data_stored":118545149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T01:57:48.036 INFO:tasks.rados.rados.0.smithi079.stdout:566: done (1 left) 2024-04-24T01:57:48.036 INFO:tasks.rados.rados.0.smithi079.stdout:567: snap_create 2024-04-24T01:57:48.036 INFO:tasks.rados.rados.0.smithi079.stdout:565: finishing write tid 1 to smithi07929762-22 2024-04-24T01:57:48.036 INFO:tasks.rados.rados.0.smithi079.stdout:565: finishing write tid 2 to smithi07929762-22 2024-04-24T01:57:48.036 INFO:tasks.rados.rados.0.smithi079.stdout:565: finishing write tid 3 to smithi07929762-22 2024-04-24T01:57:48.036 INFO:tasks.rados.rados.0.smithi079.stdout:565: finishing write tid 4 to smithi07929762-22 2024-04-24T01:57:48.036 INFO:tasks.rados.rados.0.smithi079.stdout:565: oid 22 updating version 0 to 86 2024-04-24T01:57:48.037 INFO:tasks.rados.rados.0.smithi079.stdout:565: oid 22 version 86 is already newer than 86 2024-04-24T01:57:48.037 INFO:tasks.rados.rados.0.smithi079.stdout:565: oid 22 version 86 is already newer than 86 2024-04-24T01:57:48.037 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 22 v 86 (ObjNum 277 snap 72 seq_num 277) dirty exists 2024-04-24T01:57:48.037 INFO:tasks.rados.rados.0.smithi079.stdout:565: left oid 22 (ObjNum 277 snap 72 seq_num 277) 2024-04-24T01:57:48.541 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T01:57:48.645 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:48.644+0000 7f90714ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:48.649 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:48.649+0000 7f90714ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:49.034 INFO:tasks.rados.rados.0.smithi079.stdout:565: done (1 left) 2024-04-24T01:57:49.035 INFO:tasks.rados.rados.0.smithi079.stdout:567: done (0 left) 2024-04-24T01:57:49.035 INFO:tasks.rados.rados.0.smithi079.stdout:568: snap_create 2024-04-24T01:57:50.036 INFO:tasks.rados.rados.0.smithi079.stdout:568: done (0 left) 2024-04-24T01:57:50.036 INFO:tasks.rados.rados.0.smithi079.stdout:569: snap_remove snap 51 2024-04-24T01:57:50.741 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T01:57:50.857 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:50.854+0000 7fdab195d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:50.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:50.860+0000 7fdab195d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:51.042 INFO:tasks.rados.rados.0.smithi079.stdout:569: done (0 left) 2024-04-24T01:57:51.042 INFO:tasks.rados.rados.0.smithi079.stdout:570: write oid 46 current snap is 74 2024-04-24T01:57:51.042 INFO:tasks.rados.rados.0.smithi079.stdout:570: seq_num 278 ranges {672346=707930,2134700=537646} 2024-04-24T01:57:51.049 INFO:tasks.rados.rados.0.smithi079.stdout:570: writing smithi07929762-46 from 672346 to 1380276 tid 1 2024-04-24T01:57:51.057 INFO:tasks.rados.rados.0.smithi079.stdout:570: writing smithi07929762-46 from 2134700 to 2672346 tid 2 2024-04-24T01:57:51.057 INFO:tasks.rados.rados.0.smithi079.stdout:571: snap_remove snap 57 2024-04-24T01:57:52.046 INFO:tasks.rados.rados.0.smithi079.stdout:571: done (1 left) 2024-04-24T01:57:52.046 INFO:tasks.rados.rados.0.smithi079.stdout:572: delete oid 37 current snap is 74 2024-04-24T01:57:52.046 INFO:tasks.rados.rados.0.smithi079.stdout:570: finishing write tid 1 to smithi07929762-46 2024-04-24T01:57:52.046 INFO:tasks.rados.rados.0.smithi079.stdout:570: finishing write tid 2 to smithi07929762-46 2024-04-24T01:57:52.046 INFO:tasks.rados.rados.0.smithi079.stdout:570: finishing write tid 3 to smithi07929762-46 2024-04-24T01:57:52.046 INFO:tasks.rados.rados.0.smithi079.stdout:570: finishing write tid 4 to smithi07929762-46 2024-04-24T01:57:52.046 INFO:tasks.rados.rados.0.smithi079.stdout:570: oid 46 updating version 0 to 127 2024-04-24T01:57:52.046 INFO:tasks.rados.rados.0.smithi079.stdout:570: oid 46 version 127 is already newer than 127 2024-04-24T01:57:52.046 INFO:tasks.rados.rados.0.smithi079.stdout:570: oid 46 version 127 is already newer than 127 2024-04-24T01:57:52.046 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 46 v 127 (ObjNum 278 snap 74 seq_num 278) dirty exists 2024-04-24T01:57:52.046 INFO:tasks.rados.rados.0.smithi079.stdout:570: left oid 46 (ObjNum 278 snap 74 seq_num 278) 2024-04-24T01:57:52.115 INFO:tasks.rados.rados.0.smithi079.stdout:570: done (1 left) 2024-04-24T01:57:52.115 INFO:tasks.rados.rados.0.smithi079.stdout:572: done (0 left) 2024-04-24T01:57:52.115 INFO:tasks.rados.rados.0.smithi079.stdout:573: delete oid 48 current snap is 74 2024-04-24T01:57:52.149 INFO:tasks.rados.rados.0.smithi079.stdout:573: done (0 left) 2024-04-24T01:57:52.149 INFO:tasks.rados.rados.0.smithi079.stdout:574: read oid 9 snap -1 2024-04-24T01:57:52.149 INFO:tasks.rados.rados.0.smithi079.stdout:574: expect (ObjNum 243 snap 67 seq_num 243) 2024-04-24T01:57:52.149 INFO:tasks.rados.rados.0.smithi079.stdout:575: snap_create 2024-04-24T01:57:52.245 INFO:tasks.rados.rados.0.smithi079.stdout:574: expect (ObjNum 243 snap 67 seq_num 243) 2024-04-24T01:57:52.938 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-04-24T01:57:52.938 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-04-24T01:57:52.939 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-04-24T01:57:52.939 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-04-24T01:57:52.956 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T01:57:53.048 INFO:tasks.rados.rados.0.smithi079.stdout:574: done (1 left) 2024-04-24T01:57:53.048 INFO:tasks.rados.rados.0.smithi079.stdout:575: done (0 left) 2024-04-24T01:57:53.049 INFO:tasks.rados.rados.0.smithi079.stdout:576: write oid 45 current snap is 75 2024-04-24T01:57:53.049 INFO:tasks.rados.rados.0.smithi079.stdout:576: seq_num 281 ranges {722767=533677,1793793=461082} 2024-04-24T01:57:53.056 INFO:tasks.rados.rados.0.smithi079.stdout:576: writing smithi07929762-45 from 722767 to 1256444 tid 1 2024-04-24T01:57:53.057 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:53.049+0000 7f5cb379d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:53.063 INFO:tasks.rados.rados.0.smithi079.stdout:576: writing smithi07929762-45 from 1793793 to 2254875 tid 2 2024-04-24T01:57:53.063 INFO:tasks.rados.rados.0.smithi079.stdout:577: read oid 2 snap 67 2024-04-24T01:57:53.063 INFO:tasks.rados.rados.0.smithi079.stdout:577: expect deleted 2024-04-24T01:57:53.064 INFO:tasks.rados.rados.0.smithi079.stdout:578: snap_remove snap 68 2024-04-24T01:57:53.064 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:53.063+0000 7f5cb379d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:53.084 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T01:57:53.084 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T01:57:53.084 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T01:57:53.084 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T01:57:53.085 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T01:57:53.085 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T01:57:53.085 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T01:57:53.085 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T01:57:53.085 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T01:57:53.085 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T01:57:53.085 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T01:57:53.085 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T01:57:53.087 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T01:57:53.088 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T01:57:53.088 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-04-24T01:57:53.097 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:53.095+0000 7fa59c35d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:53.101 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:53.100+0000 7fa59c35d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:53.208 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:53.202+0000 7fc9c6f1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:53.212 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:53.211+0000 7fc9c6f1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:53.228 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T01:57:53.228 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T01:57:53.228 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T01:57:53.228 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T01:57:53.228 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T01:57:53.228 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T01:57:53.228 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T01:57:53.228 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T01:57:53.228 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T01:57:53.229 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T01:57:53.229 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T01:57:53.229 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T01:57:53.231 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T01:57:53.232 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T01:57:53.232 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-04-24T01:57:53.383 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:53.378+0000 7f6c2bc6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:53.387 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:53.386+0000 7f6c2bc6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:53.403 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T01:57:53.403 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T01:57:53.403 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T01:57:53.403 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T01:57:53.403 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T01:57:53.403 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T01:57:53.403 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T01:57:53.403 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T01:57:53.403 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T01:57:53.403 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T01:57:53.404 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T01:57:53.404 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T01:57:53.406 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T01:57:53.406 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T01:57:53.406 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-04-24T01:57:53.514 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:53.508+0000 7f5b6575d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:53.518 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:53.517+0000 7f5b6575d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:53.534 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T01:57:53.534 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T01:57:53.534 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T01:57:53.534 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T01:57:53.534 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T01:57:53.534 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T01:57:53.534 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T01:57:53.534 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T01:57:53.534 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T01:57:53.535 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T01:57:53.535 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T01:57:53.535 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T01:57:53.537 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T01:57:53.537 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T01:57:54.052 INFO:tasks.rados.rados.0.smithi079.stdout:578: done (2 left) 2024-04-24T01:57:54.052 INFO:tasks.rados.rados.0.smithi079.stdout:579: snap_remove snap 74 2024-04-24T01:57:55.056 INFO:tasks.rados.rados.0.smithi079.stdout:579: done (2 left) 2024-04-24T01:57:55.056 INFO:tasks.rados.rados.0.smithi079.stdout:580: write oid 19 current snap is 75 2024-04-24T01:57:55.056 INFO:tasks.rados.rados.0.smithi079.stdout:580: seq_num 282 ranges {739574=725279,1968628=721093} 2024-04-24T01:57:55.064 INFO:tasks.rados.rados.0.smithi079.stdout:580: writing smithi07929762-19 from 739574 to 1464853 tid 1 2024-04-24T01:57:55.071 INFO:tasks.rados.rados.0.smithi079.stdout:580: writing smithi07929762-19 from 1968628 to 2689721 tid 2 2024-04-24T01:57:55.071 INFO:tasks.rados.rados.0.smithi079.stdout:581: write oid 16 current snap is 75 2024-04-24T01:57:55.071 INFO:tasks.rados.rados.0.smithi079.stdout:581: seq_num 283 ranges {756381=400528,1910757=464751} 2024-04-24T01:57:55.077 INFO:tasks.rados.rados.0.smithi079.stdout:581: writing smithi07929762-16 from 756381 to 1156909 tid 1 2024-04-24T01:57:55.086 INFO:tasks.rados.rados.0.smithi079.stdout:581: writing smithi07929762-16 from 1910757 to 2375508 tid 2 2024-04-24T01:57:55.086 INFO:tasks.rados.rados.0.smithi079.stdout:576: finishing write tid 1 to smithi07929762-45 2024-04-24T01:57:55.086 INFO:tasks.rados.rados.0.smithi079.stdout:576: finishing write tid 2 to smithi07929762-45 2024-04-24T01:57:55.086 INFO:tasks.rados.rados.0.smithi079.stdout:576: finishing write tid 3 to smithi07929762-45 2024-04-24T01:57:55.086 INFO:tasks.rados.rados.0.smithi079.stdout:576: finishing write tid 4 to smithi07929762-45 2024-04-24T01:57:55.086 INFO:tasks.rados.rados.0.smithi079.stdout:576: oid 45 updating version 0 to 160 2024-04-24T01:57:55.086 INFO:tasks.rados.rados.0.smithi079.stdout:576: oid 45 version 160 is already newer than 160 2024-04-24T01:57:55.086 INFO:tasks.rados.rados.0.smithi079.stdout:576: oid 45 version 160 is already newer than 160 2024-04-24T01:57:55.086 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 45 v 160 (ObjNum 281 snap 75 seq_num 281) dirty exists 2024-04-24T01:57:55.087 INFO:tasks.rados.rados.0.smithi079.stdout:576: left oid 45 (ObjNum 281 snap 75 seq_num 281) 2024-04-24T01:57:55.087 INFO:tasks.rados.rados.0.smithi079.stdout:582: read oid 18 snap 53 2024-04-24T01:57:55.087 INFO:tasks.rados.rados.0.smithi079.stdout:582: expect (ObjNum 185 snap 34 seq_num 185) 2024-04-24T01:57:55.087 INFO:tasks.rados.rados.0.smithi079.stdout:576: done (4 left) 2024-04-24T01:57:55.087 INFO:tasks.rados.rados.0.smithi079.stdout:577: done (3 left) 2024-04-24T01:57:55.087 INFO:tasks.rados.rados.0.smithi079.stdout:583: write oid 48 current snap is 75 2024-04-24T01:57:55.087 INFO:tasks.rados.rados.0.smithi079.stdout:583: seq_num 284 ranges {773188=475777,1969239=724762} 2024-04-24T01:57:55.091 INFO:tasks.rados.rados.0.smithi079.stdout:583: writing smithi07929762-48 from 773188 to 1248965 tid 1 2024-04-24T01:57:55.102 INFO:tasks.rados.rados.0.smithi079.stdout:583: writing smithi07929762-48 from 1969239 to 2694001 tid 2 2024-04-24T01:57:55.102 INFO:tasks.rados.rados.0.smithi079.stdout:584: write (excl) oid 4 current snap is 75 2024-04-24T01:57:55.102 INFO:tasks.rados.rados.0.smithi079.stdout:584: seq_num 285 ranges {789995=551026,2027721=468420} 2024-04-24T01:57:55.111 INFO:tasks.rados.rados.0.smithi079.stdout:584: writing smithi07929762-4 from 789995 to 1341021 tid 1 2024-04-24T01:57:55.120 INFO:tasks.rados.rados.0.smithi079.stdout:584: writing smithi07929762-4 from 2027721 to 2496141 tid 2 2024-04-24T01:57:55.121 INFO:tasks.rados.rados.0.smithi079.stdout:585: snap_remove snap 42 2024-04-24T01:57:55.238 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T01:57:55.344 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:55.342+0000 7fcdffe6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:55.348 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:55.347+0000 7fcdffe6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:56.035 INFO:tasks.rados.rados.0.smithi079.stdout:585: done (5 left) 2024-04-24T01:57:56.035 INFO:tasks.rados.rados.0.smithi079.stdout:586: write (excl) oid 2 current snap is 75 2024-04-24T01:57:56.035 INFO:tasks.rados.rados.0.smithi079.stdout:581: finishing write tid 1 to smithi07929762-16 2024-04-24T01:57:56.035 INFO:tasks.rados.rados.0.smithi079.stdout:586: seq_num 286 ranges {406802=626275,1686203=728431} 2024-04-24T01:57:56.040 INFO:tasks.rados.rados.0.smithi079.stdout:586: writing smithi07929762-2 from 406802 to 1033077 tid 1 2024-04-24T01:57:56.049 INFO:tasks.rados.rados.0.smithi079.stdout:586: writing smithi07929762-2 from 1686203 to 2414634 tid 2 2024-04-24T01:57:56.049 INFO:tasks.rados.rados.0.smithi079.stdout:587: snap_create 2024-04-24T01:57:56.049 INFO:tasks.rados.rados.0.smithi079.stdout:581: finishing write tid 2 to smithi07929762-16 2024-04-24T01:57:56.049 INFO:tasks.rados.rados.0.smithi079.stdout:581: finishing write tid 3 to smithi07929762-16 2024-04-24T01:57:56.049 INFO:tasks.rados.rados.0.smithi079.stdout:584: finishing write tid 1 to smithi07929762-4 2024-04-24T01:57:56.049 INFO:tasks.rados.rados.0.smithi079.stdout:581: finishing write tid 4 to smithi07929762-16 2024-04-24T01:57:56.049 INFO:tasks.rados.rados.0.smithi079.stdout:581: oid 16 updating version 0 to 250 2024-04-24T01:57:56.049 INFO:tasks.rados.rados.0.smithi079.stdout:581: oid 16 version 250 is already newer than 250 2024-04-24T01:57:56.049 INFO:tasks.rados.rados.0.smithi079.stdout:581: oid 16 version 250 is already newer than 250 2024-04-24T01:57:56.049 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 16 v 250 (ObjNum 283 snap 75 seq_num 283) dirty exists 2024-04-24T01:57:56.049 INFO:tasks.rados.rados.0.smithi079.stdout:581: left oid 16 (ObjNum 283 snap 75 seq_num 283) 2024-04-24T01:57:56.050 INFO:tasks.rados.rados.0.smithi079.stdout:580: finishing write tid 1 to smithi07929762-19 2024-04-24T01:57:56.050 INFO:tasks.rados.rados.0.smithi079.stdout:580: finishing write tid 2 to smithi07929762-19 2024-04-24T01:57:56.050 INFO:tasks.rados.rados.0.smithi079.stdout:580: finishing write tid 3 to smithi07929762-19 2024-04-24T01:57:56.050 INFO:tasks.rados.rados.0.smithi079.stdout:580: finishing write tid 4 to smithi07929762-19 2024-04-24T01:57:56.050 INFO:tasks.rados.rados.0.smithi079.stdout:580: oid 19 updating version 0 to 149 2024-04-24T01:57:56.050 INFO:tasks.rados.rados.0.smithi079.stdout:580: oid 19 version 149 is already newer than 149 2024-04-24T01:57:56.050 INFO:tasks.rados.rados.0.smithi079.stdout:580: oid 19 version 149 is already newer than 149 2024-04-24T01:57:56.050 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 19 v 149 (ObjNum 282 snap 75 seq_num 282) dirty exists 2024-04-24T01:57:56.050 INFO:tasks.rados.rados.0.smithi079.stdout:580: left oid 19 (ObjNum 282 snap 75 seq_num 282) 2024-04-24T01:57:56.050 INFO:tasks.rados.rados.0.smithi079.stdout:584: finishing write tid 2 to smithi07929762-4 2024-04-24T01:57:56.050 INFO:tasks.rados.rados.0.smithi079.stdout:583: finishing write tid 1 to smithi07929762-48 2024-04-24T01:57:56.050 INFO:tasks.rados.rados.0.smithi079.stdout:582: expect (ObjNum 185 snap 34 seq_num 185) 2024-04-24T01:57:56.236 INFO:tasks.rados.rados.0.smithi079.stdout:584: finishing write tid 3 to smithi07929762-4 2024-04-24T01:57:56.236 INFO:tasks.rados.rados.0.smithi079.stdout:583: finishing write tid 2 to smithi07929762-48 2024-04-24T01:57:56.236 INFO:tasks.rados.rados.0.smithi079.stdout:584: finishing write tid 4 to smithi07929762-4 2024-04-24T01:57:56.236 INFO:tasks.rados.rados.0.smithi079.stdout:584: oid 4 updating version 0 to 38 2024-04-24T01:57:56.236 INFO:tasks.rados.rados.0.smithi079.stdout:584: oid 4 updating version 38 to 40 2024-04-24T01:57:56.236 INFO:tasks.rados.rados.0.smithi079.stdout:584: oid 4 version 40 is already newer than 40 2024-04-24T01:57:56.237 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 4 v 40 (ObjNum 285 snap 75 seq_num 285) dirty exists 2024-04-24T01:57:56.237 INFO:tasks.rados.rados.0.smithi079.stdout:584: left oid 4 (ObjNum 285 snap 75 seq_num 285) 2024-04-24T01:57:56.237 INFO:tasks.rados.rados.0.smithi079.stdout:583: finishing write tid 3 to smithi07929762-48 2024-04-24T01:57:56.237 INFO:tasks.rados.rados.0.smithi079.stdout:583: finishing write tid 4 to smithi07929762-48 2024-04-24T01:57:56.237 INFO:tasks.rados.rados.0.smithi079.stdout:583: oid 48 updating version 0 to 90 2024-04-24T01:57:56.237 INFO:tasks.rados.rados.0.smithi079.stdout:583: oid 48 version 90 is already newer than 90 2024-04-24T01:57:56.237 INFO:tasks.rados.rados.0.smithi079.stdout:583: oid 48 version 90 is already newer than 90 2024-04-24T01:57:56.237 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 48 v 90 (ObjNum 284 snap 75 seq_num 284) dirty exists 2024-04-24T01:57:56.237 INFO:tasks.rados.rados.0.smithi079.stdout:583: left oid 48 (ObjNum 284 snap 75 seq_num 284) 2024-04-24T01:57:56.237 INFO:tasks.rados.rados.0.smithi079.stdout:586: finishing write tid 1 to smithi07929762-2 2024-04-24T01:57:56.237 INFO:tasks.rados.rados.0.smithi079.stdout:586: finishing write tid 2 to smithi07929762-2 2024-04-24T01:57:56.237 INFO:tasks.rados.rados.0.smithi079.stdout:586: finishing write tid 3 to smithi07929762-2 2024-04-24T01:57:56.237 INFO:tasks.rados.rados.0.smithi079.stdout:586: finishing write tid 4 to smithi07929762-2 2024-04-24T01:57:56.237 INFO:tasks.rados.rados.0.smithi079.stdout:586: oid 2 updating version 0 to 47 2024-04-24T01:57:56.237 INFO:tasks.rados.rados.0.smithi079.stdout:586: oid 2 version 47 is already newer than 47 2024-04-24T01:57:56.238 INFO:tasks.rados.rados.0.smithi079.stdout:586: oid 2 version 47 is already newer than 45 2024-04-24T01:57:56.238 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 2 v 47 (ObjNum 286 snap 75 seq_num 286) dirty exists 2024-04-24T01:57:56.238 INFO:tasks.rados.rados.0.smithi079.stdout:586: left oid 2 (ObjNum 286 snap 75 seq_num 286) 2024-04-24T01:57:57.041 INFO:tasks.rados.rados.0.smithi079.stdout:580: done (6 left) 2024-04-24T01:57:57.041 INFO:tasks.rados.rados.0.smithi079.stdout:581: done (5 left) 2024-04-24T01:57:57.041 INFO:tasks.rados.rados.0.smithi079.stdout:582: done (4 left) 2024-04-24T01:57:57.041 INFO:tasks.rados.rados.0.smithi079.stdout:583: done (3 left) 2024-04-24T01:57:57.041 INFO:tasks.rados.rados.0.smithi079.stdout:584: done (2 left) 2024-04-24T01:57:57.041 INFO:tasks.rados.rados.0.smithi079.stdout:586: done (1 left) 2024-04-24T01:57:57.041 INFO:tasks.rados.rados.0.smithi079.stdout:587: done (0 left) 2024-04-24T01:57:57.041 INFO:tasks.rados.rados.0.smithi079.stdout:588: write (excl) oid 2 current snap is 76 2024-04-24T01:57:57.041 INFO:tasks.rados.rados.0.smithi079.stdout:588: seq_num 287 ranges {423609=701524,1628332=472089,2648987=174622} 2024-04-24T01:57:57.047 INFO:tasks.rados.rados.0.smithi079.stdout:588: writing smithi07929762-2 from 423609 to 1125133 tid 1 2024-04-24T01:57:57.052 INFO:tasks.rados.rados.0.smithi079.stdout:588: writing smithi07929762-2 from 1628332 to 2100421 tid 2 2024-04-24T01:57:57.057 INFO:tasks.rados.rados.0.smithi079.stdout:588: writing smithi07929762-2 from 2648987 to 2823609 tid 3 2024-04-24T01:57:57.057 INFO:tasks.rados.rados.0.smithi079.stdout:589: read oid 35 snap -1 2024-04-24T01:57:57.057 INFO:tasks.rados.rados.0.smithi079.stdout:589: expect (ObjNum 188 snap 36 seq_num 188) 2024-04-24T01:57:57.057 INFO:tasks.rados.rados.0.smithi079.stdout:590: delete oid 8 current snap is 76 2024-04-24T01:57:57.123 INFO:tasks.rados.rados.0.smithi079.stdout:588: finishing write tid 1 to smithi07929762-2 2024-04-24T01:57:57.152 INFO:tasks.rados.rados.0.smithi079.stdout:590: done (2 left) 2024-04-24T01:57:57.152 INFO:tasks.rados.rados.0.smithi079.stdout:591: write (excl) oid 32 current snap is 76 2024-04-24T01:57:57.152 INFO:tasks.rados.rados.0.smithi079.stdout:591: seq_num 289 ranges {457223=568375,1461649=475758,2420186=437037} 2024-04-24T01:57:57.159 INFO:tasks.rados.rados.0.smithi079.stdout:591: writing smithi07929762-32 from 457223 to 1025598 tid 1 2024-04-24T01:57:57.165 INFO:tasks.rados.rados.0.smithi079.stdout:591: writing smithi07929762-32 from 1461649 to 1937407 tid 2 2024-04-24T01:57:57.171 INFO:tasks.rados.rados.0.smithi079.stdout:591: writing smithi07929762-32 from 2420186 to 2857223 tid 3 2024-04-24T01:57:57.171 INFO:tasks.rados.rados.0.smithi079.stdout:592: read oid 26 snap -1 2024-04-24T01:57:57.171 INFO:tasks.rados.rados.0.smithi079.stdout:592: expect (ObjNum 259 snap 70 seq_num 259) 2024-04-24T01:57:57.171 INFO:tasks.rados.rados.0.smithi079.stdout:593: write oid 30 current snap is 76 2024-04-24T01:57:57.171 INFO:tasks.rados.rados.0.smithi079.stdout:588: finishing write tid 2 to smithi07929762-2 2024-04-24T01:57:57.171 INFO:tasks.rados.rados.0.smithi079.stdout:593: seq_num 290 ranges {474030=643624,1520131=735769} 2024-04-24T01:57:57.180 INFO:tasks.rados.rados.0.smithi079.stdout:593: writing smithi07929762-30 from 474030 to 1117654 tid 1 2024-04-24T01:57:57.188 INFO:tasks.rados.rados.0.smithi079.stdout:593: writing smithi07929762-30 from 1520131 to 2255900 tid 2 2024-04-24T01:57:57.188 INFO:tasks.rados.rados.0.smithi079.stdout:594: write (excl) oid 5 current snap is 76 2024-04-24T01:57:57.188 INFO:tasks.rados.rados.0.smithi079.stdout:589: expect (ObjNum 188 snap 36 seq_num 188) 2024-04-24T01:57:57.439 INFO:tasks.rados.rados.0.smithi079.stdout:588: finishing write tid 3 to smithi07929762-2 2024-04-24T01:57:57.439 INFO:tasks.rados.rados.0.smithi079.stdout:588: finishing write tid 4 to smithi07929762-2 2024-04-24T01:57:57.440 INFO:tasks.rados.rados.0.smithi079.stdout:588: finishing write tid 5 to smithi07929762-2 2024-04-24T01:57:57.440 INFO:tasks.rados.rados.0.smithi079.stdout:588: oid 2 updating version 0 to 52 2024-04-24T01:57:57.440 INFO:tasks.rados.rados.0.smithi079.stdout:588: oid 2 version 52 is already newer than 52 2024-04-24T01:57:57.440 INFO:tasks.rados.rados.0.smithi079.stdout:588: oid 2 version 52 is already newer than 49 2024-04-24T01:57:57.440 INFO:tasks.rados.rados.0.smithi079.stdout:588: oid 2 version 52 is already newer than 52 2024-04-24T01:57:57.440 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 2 v 52 (ObjNum 287 snap 76 seq_num 287) dirty exists 2024-04-24T01:57:57.440 INFO:tasks.rados.rados.0.smithi079.stdout:588: left oid 2 (ObjNum 287 snap 76 seq_num 287) 2024-04-24T01:57:57.440 INFO:tasks.rados.rados.0.smithi079.stdout:594: seq_num 291 ranges {490837=718873,1862260=479427,2875032=15805} 2024-04-24T01:57:57.440 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T01:57:57.446 INFO:tasks.rados.rados.0.smithi079.stdout:594: writing smithi07929762-5 from 490837 to 1209710 tid 1 2024-04-24T01:57:57.452 INFO:tasks.rados.rados.0.smithi079.stdout:594: writing smithi07929762-5 from 1862260 to 2341687 tid 2 2024-04-24T01:57:57.456 INFO:tasks.rados.rados.0.smithi079.stdout:594: writing smithi07929762-5 from 2875032 to 2890837 tid 3 2024-04-24T01:57:57.456 INFO:tasks.rados.rados.0.smithi079.stdout:588: done (5 left) 2024-04-24T01:57:57.456 INFO:tasks.rados.rados.0.smithi079.stdout:589: done (4 left) 2024-04-24T01:57:57.456 INFO:tasks.rados.rados.0.smithi079.stdout:595: write (excl) oid 20 current snap is 76 2024-04-24T01:57:57.456 INFO:tasks.rados.rados.0.smithi079.stdout:592: expect (ObjNum 259 snap 70 seq_num 259) 2024-04-24T01:57:57.560 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:57.557+0000 7fc82c8ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:57.563 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:57.562+0000 7fc82c8ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:57.595 INFO:tasks.rados.rados.0.smithi079.stdout:593: finishing write tid 1 to smithi07929762-30 2024-04-24T01:57:57.595 INFO:tasks.rados.rados.0.smithi079.stdout:591: finishing write tid 1 to smithi07929762-32 2024-04-24T01:57:57.596 INFO:tasks.rados.rados.0.smithi079.stdout:593: finishing write tid 2 to smithi07929762-30 2024-04-24T01:57:57.596 INFO:tasks.rados.rados.0.smithi079.stdout:593: finishing write tid 3 to smithi07929762-30 2024-04-24T01:57:57.596 INFO:tasks.rados.rados.0.smithi079.stdout:591: finishing write tid 2 to smithi07929762-32 2024-04-24T01:57:57.596 INFO:tasks.rados.rados.0.smithi079.stdout:593: finishing write tid 4 to smithi07929762-30 2024-04-24T01:57:57.596 INFO:tasks.rados.rados.0.smithi079.stdout:593: oid 30 updating version 0 to 65 2024-04-24T01:57:57.596 INFO:tasks.rados.rados.0.smithi079.stdout:593: oid 30 version 65 is already newer than 65 2024-04-24T01:57:57.596 INFO:tasks.rados.rados.0.smithi079.stdout:593: oid 30 version 65 is already newer than 65 2024-04-24T01:57:57.596 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 30 v 65 (ObjNum 290 snap 76 seq_num 290) dirty exists 2024-04-24T01:57:57.596 INFO:tasks.rados.rados.0.smithi079.stdout:593: left oid 30 (ObjNum 290 snap 76 seq_num 290) 2024-04-24T01:57:57.596 INFO:tasks.rados.rados.0.smithi079.stdout:591: finishing write tid 3 to smithi07929762-32 2024-04-24T01:57:57.596 INFO:tasks.rados.rados.0.smithi079.stdout:591: finishing write tid 4 to smithi07929762-32 2024-04-24T01:57:57.596 INFO:tasks.rados.rados.0.smithi079.stdout:591: finishing write tid 5 to smithi07929762-32 2024-04-24T01:57:57.596 INFO:tasks.rados.rados.0.smithi079.stdout:591: oid 32 updating version 0 to 99 2024-04-24T01:57:57.596 INFO:tasks.rados.rados.0.smithi079.stdout:591: oid 32 version 99 is already newer than 99 2024-04-24T01:57:57.596 INFO:tasks.rados.rados.0.smithi079.stdout:591: oid 32 version 99 is already newer than 99 2024-04-24T01:57:57.596 INFO:tasks.rados.rados.0.smithi079.stdout:591: oid 32 version 99 is already newer than 96 2024-04-24T01:57:57.597 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 32 v 99 (ObjNum 289 snap 76 seq_num 289) dirty exists 2024-04-24T01:57:57.597 INFO:tasks.rados.rados.0.smithi079.stdout:591: left oid 32 (ObjNum 289 snap 76 seq_num 289) 2024-04-24T01:57:57.597 INFO:tasks.rados.rados.0.smithi079.stdout:595: seq_num 292 ranges {507644=794122,1920742=623085} 2024-04-24T01:57:57.602 INFO:tasks.rados.rados.0.smithi079.stdout:595: writing smithi07929762-20 from 507644 to 1301766 tid 1 2024-04-24T01:57:57.613 INFO:tasks.rados.rados.0.smithi079.stdout:595: writing smithi07929762-20 from 1920742 to 2543827 tid 2 2024-04-24T01:57:57.613 INFO:tasks.rados.rados.0.smithi079.stdout:591: done (4 left) 2024-04-24T01:57:57.613 INFO:tasks.rados.rados.0.smithi079.stdout:592: done (3 left) 2024-04-24T01:57:57.613 INFO:tasks.rados.rados.0.smithi079.stdout:593: done (2 left) 2024-04-24T01:57:57.613 INFO:tasks.rados.rados.0.smithi079.stdout:596: read oid 47 snap 63 2024-04-24T01:57:57.613 INFO:tasks.rados.rados.0.smithi079.stdout:596: expect (ObjNum 193 snap 38 seq_num 193) 2024-04-24T01:57:57.613 INFO:tasks.rados.rados.0.smithi079.stdout:597: delete oid 46 current snap is 76 2024-04-24T01:57:57.613 INFO:tasks.rados.rados.0.smithi079.stdout:594: finishing write tid 1 to smithi07929762-5 2024-04-24T01:57:57.613 INFO:tasks.rados.rados.0.smithi079.stdout:594: finishing write tid 2 to smithi07929762-5 2024-04-24T01:57:57.613 INFO:tasks.rados.rados.0.smithi079.stdout:594: finishing write tid 3 to smithi07929762-5 2024-04-24T01:57:57.613 INFO:tasks.rados.rados.0.smithi079.stdout:594: finishing write tid 4 to smithi07929762-5 2024-04-24T01:57:57.613 INFO:tasks.rados.rados.0.smithi079.stdout:594: finishing write tid 5 to smithi07929762-5 2024-04-24T01:57:57.613 INFO:tasks.rados.rados.0.smithi079.stdout:594: oid 5 updating version 0 to 135 2024-04-24T01:57:57.613 INFO:tasks.rados.rados.0.smithi079.stdout:594: oid 5 updating version 135 to 138 2024-04-24T01:57:57.614 INFO:tasks.rados.rados.0.smithi079.stdout:594: oid 5 version 138 is already newer than 138 2024-04-24T01:57:57.614 INFO:tasks.rados.rados.0.smithi079.stdout:594: oid 5 version 138 is already newer than 138 2024-04-24T01:57:57.614 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 5 v 138 (ObjNum 291 snap 76 seq_num 291) dirty exists 2024-04-24T01:57:57.614 INFO:tasks.rados.rados.0.smithi079.stdout:594: left oid 5 (ObjNum 291 snap 76 seq_num 291) 2024-04-24T01:57:57.641 INFO:tasks.rados.rados.0.smithi079.stdout:596: expect (ObjNum 193 snap 38 seq_num 193) 2024-04-24T01:57:57.828 INFO:tasks.rados.rados.0.smithi079.stdout:595: finishing write tid 1 to smithi07929762-20 2024-04-24T01:57:57.828 INFO:tasks.rados.rados.0.smithi079.stdout:595: finishing write tid 2 to smithi07929762-20 2024-04-24T01:57:57.828 INFO:tasks.rados.rados.0.smithi079.stdout:595: finishing write tid 3 to smithi07929762-20 2024-04-24T01:57:57.828 INFO:tasks.rados.rados.0.smithi079.stdout:595: finishing write tid 4 to smithi07929762-20 2024-04-24T01:57:57.828 INFO:tasks.rados.rados.0.smithi079.stdout:595: oid 20 updating version 0 to 210 2024-04-24T01:57:57.828 INFO:tasks.rados.rados.0.smithi079.stdout:595: oid 20 version 210 is already newer than 210 2024-04-24T01:57:57.828 INFO:tasks.rados.rados.0.smithi079.stdout:595: oid 20 version 210 is already newer than 208 2024-04-24T01:57:57.828 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 20 v 210 (ObjNum 292 snap 76 seq_num 292) dirty exists 2024-04-24T01:57:57.829 INFO:tasks.rados.rados.0.smithi079.stdout:595: left oid 20 (ObjNum 292 snap 76 seq_num 292) 2024-04-24T01:57:57.829 INFO:tasks.rados.rados.0.smithi079.stdout:594: done (3 left) 2024-04-24T01:57:57.829 INFO:tasks.rados.rados.0.smithi079.stdout:595: done (2 left) 2024-04-24T01:57:57.829 INFO:tasks.rados.rados.0.smithi079.stdout:596: done (1 left) 2024-04-24T01:57:57.829 INFO:tasks.rados.rados.0.smithi079.stdout:597: done (0 left) 2024-04-24T01:57:57.829 INFO:tasks.rados.rados.0.smithi079.stdout:598: snap_remove snap 73 2024-04-24T01:57:58.060 INFO:tasks.rados.rados.0.smithi079.stdout:598: done (0 left) 2024-04-24T01:57:58.060 INFO:tasks.rados.rados.0.smithi079.stdout:599: snap_create 2024-04-24T01:57:58.538 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-04-24T01:57:58.538 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-04-24T01:57:58.538 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-04-24T01:57:58.539 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-04-24T01:57:58.648 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:58.642+0000 7efec715d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:58.651 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:58.650+0000 7efec715d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:58.665 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T01:57:58.665 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T01:57:58.665 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T01:57:58.665 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T01:57:58.665 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T01:57:58.665 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T01:57:58.665 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T01:57:58.665 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T01:57:58.665 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T01:57:58.666 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T01:57:58.666 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T01:57:58.666 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T01:57:58.668 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T01:57:58.669 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T01:57:58.669 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-04-24T01:57:58.776 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:58.770+0000 7fbc756ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:58.779 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:58.779+0000 7fbc756ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:58.794 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T01:57:58.794 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T01:57:58.794 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T01:57:58.794 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T01:57:58.794 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T01:57:58.794 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T01:57:58.794 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T01:57:58.794 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T01:57:58.794 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T01:57:58.794 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T01:57:58.794 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T01:57:58.795 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T01:57:58.797 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T01:57:58.797 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T01:57:58.797 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-04-24T01:57:58.904 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:58.899+0000 7f68c14ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:58.908 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:58.907+0000 7f68c14ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:58.924 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T01:57:58.925 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T01:57:58.925 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T01:57:58.925 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T01:57:58.925 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T01:57:58.925 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T01:57:58.925 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T01:57:58.925 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T01:57:58.925 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T01:57:58.925 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T01:57:58.925 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T01:57:58.925 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T01:57:58.928 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T01:57:58.928 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T01:57:58.928 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-04-24T01:57:59.036 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:59.030+0000 7f51e0f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:59.040 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:59.039+0000 7f51e0f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:59.056 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T01:57:59.056 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T01:57:59.056 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T01:57:59.056 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T01:57:59.056 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T01:57:59.056 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T01:57:59.056 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T01:57:59.057 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T01:57:59.057 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T01:57:59.057 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T01:57:59.057 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T01:57:59.057 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T01:57:59.059 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T01:57:59.059 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T01:57:59.070 INFO:tasks.rados.rados.0.smithi079.stdout:599: done (0 left) 2024-04-24T01:57:59.070 INFO:tasks.rados.rados.0.smithi079.stdout:600: snap_create 2024-04-24T01:57:59.697 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T01:57:59.802 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:59.800+0000 7fc6ff46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:57:59.806 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:57:59.805+0000 7fc6ff46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:00.070 INFO:tasks.rados.rados.0.smithi079.stdout:600: done (0 left) 2024-04-24T01:58:00.070 INFO:tasks.rados.rados.0.smithi079.stdout:601: write oid 29 current snap is 78 2024-04-24T01:58:00.070 INFO:tasks.rados.rados.0.smithi079.stdout:601: seq_num 294 ranges {541258=544620,1637706=626754,2840948=100310} 2024-04-24T01:58:00.075 INFO:tasks.rados.rados.0.smithi079.stdout:601: writing smithi07929762-29 from 541258 to 1085878 tid 1 2024-04-24T01:58:00.083 INFO:tasks.rados.rados.0.smithi079.stdout:601: writing smithi07929762-29 from 1637706 to 2264460 tid 2 2024-04-24T01:58:00.087 INFO:tasks.rados.rados.0.smithi079.stdout:601: writing smithi07929762-29 from 2840948 to 2941258 tid 3 2024-04-24T01:58:00.087 INFO:tasks.rados.rados.0.smithi079.stdout:602: write (excl) oid 19 current snap is 78 2024-04-24T01:58:00.088 INFO:tasks.rados.rados.0.smithi079.stdout:602: seq_num 295 ranges {558065=619869,1579835=486765,2468371=489694} 2024-04-24T01:58:00.094 INFO:tasks.rados.rados.0.smithi079.stdout:602: writing smithi07929762-19 from 558065 to 1177934 tid 1 2024-04-24T01:58:00.099 INFO:tasks.rados.rados.0.smithi079.stdout:602: writing smithi07929762-19 from 1579835 to 2066600 tid 2 2024-04-24T01:58:00.105 INFO:tasks.rados.rados.0.smithi079.stdout:602: writing smithi07929762-19 from 2468371 to 2958065 tid 3 2024-04-24T01:58:00.105 INFO:tasks.rados.rados.0.smithi079.stdout:603: write (excl) oid 31 current snap is 78 2024-04-24T01:58:00.105 INFO:tasks.rados.rados.0.smithi079.stdout:603: seq_num 296 ranges {574872=695118,2038317=630423} 2024-04-24T01:58:00.111 INFO:tasks.rados.rados.0.smithi079.stdout:603: writing smithi07929762-31 from 574872 to 1269990 tid 1 2024-04-24T01:58:00.120 INFO:tasks.rados.rados.0.smithi079.stdout:603: writing smithi07929762-31 from 2038317 to 2668740 tid 2 2024-04-24T01:58:00.120 INFO:tasks.rados.rados.0.smithi079.stdout:604: read oid 22 snap -1 2024-04-24T01:58:00.120 INFO:tasks.rados.rados.0.smithi079.stdout:604: expect (ObjNum 277 snap 72 seq_num 277) 2024-04-24T01:58:00.120 INFO:tasks.rados.rados.0.smithi079.stdout:605: read oid 12 snap 67 2024-04-24T01:58:00.120 INFO:tasks.rados.rados.0.smithi079.stdout:605: expect deleted 2024-04-24T01:58:00.120 INFO:tasks.rados.rados.0.smithi079.stdout:606: read oid 9 snap -1 2024-04-24T01:58:00.121 INFO:tasks.rados.rados.0.smithi079.stdout:606: expect (ObjNum 243 snap 67 seq_num 243) 2024-04-24T01:58:00.121 INFO:tasks.rados.rados.0.smithi079.stdout:607: write (excl) oid 24 current snap is 78 2024-04-24T01:58:00.121 INFO:tasks.rados.rados.0.smithi079.stdout:607: seq_num 297 ranges {591679=486720,1813152=490434} 2024-04-24T01:58:00.126 INFO:tasks.rados.rados.0.smithi079.stdout:607: writing smithi07929762-24 from 591679 to 1078399 tid 1 2024-04-24T01:58:00.134 INFO:tasks.rados.rados.0.smithi079.stdout:607: writing smithi07929762-24 from 1813152 to 2303586 tid 2 2024-04-24T01:58:00.134 INFO:tasks.rados.rados.0.smithi079.stdout:608: read oid 41 snap -1 2024-04-24T01:58:00.134 INFO:tasks.rados.rados.0.smithi079.stdout:608: expect deleted 2024-04-24T01:58:00.134 INFO:tasks.rados.rados.0.smithi079.stdout:609: read oid 17 snap -1 2024-04-24T01:58:00.134 INFO:tasks.rados.rados.0.smithi079.stdout:609: expect (ObjNum 269 snap 70 seq_num 269) 2024-04-24T01:58:00.134 INFO:tasks.rados.rados.0.smithi079.stdout:610: delete oid 40 current snap is 78 2024-04-24T01:58:00.163 INFO:tasks.rados.rados.0.smithi079.stdout:604: expect (ObjNum 277 snap 72 seq_num 277) 2024-04-24T01:58:00.326 INFO:tasks.rados.rados.0.smithi079.stdout:606: expect (ObjNum 243 snap 67 seq_num 243) 2024-04-24T01:58:00.469 INFO:tasks.rados.rados.0.smithi079.stdout:609: expect (ObjNum 269 snap 70 seq_num 269) 2024-04-24T01:58:00.697 INFO:tasks.rados.rados.0.smithi079.stdout:601: finishing write tid 1 to smithi07929762-29 2024-04-24T01:58:00.697 INFO:tasks.rados.rados.0.smithi079.stdout:601: finishing write tid 2 to smithi07929762-29 2024-04-24T01:58:00.697 INFO:tasks.rados.rados.0.smithi079.stdout:604: done (9 left) 2024-04-24T01:58:00.697 INFO:tasks.rados.rados.0.smithi079.stdout:605: done (8 left) 2024-04-24T01:58:00.697 INFO:tasks.rados.rados.0.smithi079.stdout:606: done (7 left) 2024-04-24T01:58:00.697 INFO:tasks.rados.rados.0.smithi079.stdout:608: done (6 left) 2024-04-24T01:58:00.698 INFO:tasks.rados.rados.0.smithi079.stdout:609: done (5 left) 2024-04-24T01:58:00.698 INFO:tasks.rados.rados.0.smithi079.stdout:610: done (4 left) 2024-04-24T01:58:00.698 INFO:tasks.rados.rados.0.smithi079.stdout:611: read oid 37 snap 71 2024-04-24T01:58:00.698 INFO:tasks.rados.rados.0.smithi079.stdout:611: expect deleted 2024-04-24T01:58:00.698 INFO:tasks.rados.rados.0.smithi079.stdout:612: write (excl) oid 28 current snap is 78 2024-04-24T01:58:00.698 INFO:tasks.rados.rados.0.smithi079.stdout:602: finishing write tid 1 to smithi07929762-19 2024-04-24T01:58:00.698 INFO:tasks.rados.rados.0.smithi079.stdout:601: finishing write tid 3 to smithi07929762-29 2024-04-24T01:58:00.698 INFO:tasks.rados.rados.0.smithi079.stdout:612: seq_num 299 ranges {625293=637218,1813763=494103,2978063=47230} 2024-04-24T01:58:00.704 INFO:tasks.rados.rados.0.smithi079.stdout:612: writing smithi07929762-28 from 625293 to 1262511 tid 1 2024-04-24T01:58:00.710 INFO:tasks.rados.rados.0.smithi079.stdout:612: writing smithi07929762-28 from 1813763 to 2307866 tid 2 2024-04-24T01:58:00.715 INFO:tasks.rados.rados.0.smithi079.stdout:612: writing smithi07929762-28 from 2978063 to 3025293 tid 3 2024-04-24T01:58:00.715 INFO:tasks.rados.rados.0.smithi079.stdout:613: delete oid 7 current snap is 78 2024-04-24T01:58:00.715 INFO:tasks.rados.rados.0.smithi079.stdout:601: finishing write tid 4 to smithi07929762-29 2024-04-24T01:58:00.715 INFO:tasks.rados.rados.0.smithi079.stdout:601: finishing write tid 5 to smithi07929762-29 2024-04-24T01:58:00.715 INFO:tasks.rados.rados.0.smithi079.stdout:601: oid 29 updating version 0 to 150 2024-04-24T01:58:00.715 INFO:tasks.rados.rados.0.smithi079.stdout:601: oid 29 version 150 is already newer than 150 2024-04-24T01:58:00.715 INFO:tasks.rados.rados.0.smithi079.stdout:601: oid 29 version 150 is already newer than 150 2024-04-24T01:58:00.715 INFO:tasks.rados.rados.0.smithi079.stdout:601: oid 29 version 150 is already newer than 150 2024-04-24T01:58:00.715 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 29 v 150 (ObjNum 294 snap 78 seq_num 294) dirty exists 2024-04-24T01:58:00.715 INFO:tasks.rados.rados.0.smithi079.stdout:601: left oid 29 (ObjNum 294 snap 78 seq_num 294) 2024-04-24T01:58:00.715 INFO:tasks.rados.rados.0.smithi079.stdout:603: finishing write tid 1 to smithi07929762-31 2024-04-24T01:58:00.715 INFO:tasks.rados.rados.0.smithi079.stdout:602: finishing write tid 2 to smithi07929762-19 2024-04-24T01:58:00.715 INFO:tasks.rados.rados.0.smithi079.stdout:603: finishing write tid 2 to smithi07929762-31 2024-04-24T01:58:00.716 INFO:tasks.rados.rados.0.smithi079.stdout:602: finishing write tid 3 to smithi07929762-19 2024-04-24T01:58:00.716 INFO:tasks.rados.rados.0.smithi079.stdout:602: finishing write tid 4 to smithi07929762-19 2024-04-24T01:58:00.716 INFO:tasks.rados.rados.0.smithi079.stdout:603: finishing write tid 3 to smithi07929762-31 2024-04-24T01:58:00.716 INFO:tasks.rados.rados.0.smithi079.stdout:603: finishing write tid 4 to smithi07929762-31 2024-04-24T01:58:00.716 INFO:tasks.rados.rados.0.smithi079.stdout:603: oid 31 updating version 0 to 264 2024-04-24T01:58:00.716 INFO:tasks.rados.rados.0.smithi079.stdout:603: oid 31 updating version 264 to 266 2024-04-24T01:58:00.716 INFO:tasks.rados.rados.0.smithi079.stdout:603: oid 31 version 266 is already newer than 266 2024-04-24T01:58:00.716 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 31 v 266 (ObjNum 296 snap 78 seq_num 296) dirty exists 2024-04-24T01:58:00.716 INFO:tasks.rados.rados.0.smithi079.stdout:603: left oid 31 (ObjNum 296 snap 78 seq_num 296) 2024-04-24T01:58:00.716 INFO:tasks.rados.rados.0.smithi079.stdout:602: finishing write tid 5 to smithi07929762-19 2024-04-24T01:58:00.716 INFO:tasks.rados.rados.0.smithi079.stdout:602: oid 19 updating version 0 to 160 2024-04-24T01:58:00.716 INFO:tasks.rados.rados.0.smithi079.stdout:602: oid 19 version 160 is already newer than 157 2024-04-24T01:58:00.716 INFO:tasks.rados.rados.0.smithi079.stdout:602: oid 19 version 160 is already newer than 160 2024-04-24T01:58:00.716 INFO:tasks.rados.rados.0.smithi079.stdout:602: oid 19 version 160 is already newer than 160 2024-04-24T01:58:00.716 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 19 v 160 (ObjNum 295 snap 78 seq_num 295) dirty exists 2024-04-24T01:58:00.717 INFO:tasks.rados.rados.0.smithi079.stdout:602: left oid 19 (ObjNum 295 snap 78 seq_num 295) 2024-04-24T01:58:00.717 INFO:tasks.rados.rados.0.smithi079.stdout:607: finishing write tid 1 to smithi07929762-24 2024-04-24T01:58:00.717 INFO:tasks.rados.rados.0.smithi079.stdout:607: finishing write tid 2 to smithi07929762-24 2024-04-24T01:58:00.717 INFO:tasks.rados.rados.0.smithi079.stdout:607: finishing write tid 3 to smithi07929762-24 2024-04-24T01:58:00.717 INFO:tasks.rados.rados.0.smithi079.stdout:607: finishing write tid 4 to smithi07929762-24 2024-04-24T01:58:00.717 INFO:tasks.rados.rados.0.smithi079.stdout:607: oid 24 updating version 0 to 164 2024-04-24T01:58:00.717 INFO:tasks.rados.rados.0.smithi079.stdout:607: oid 24 version 164 is already newer than 162 2024-04-24T01:58:00.717 INFO:tasks.rados.rados.0.smithi079.stdout:607: oid 24 version 164 is already newer than 164 2024-04-24T01:58:00.717 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 24 v 164 (ObjNum 297 snap 78 seq_num 297) dirty exists 2024-04-24T01:58:00.717 INFO:tasks.rados.rados.0.smithi079.stdout:607: left oid 24 (ObjNum 297 snap 78 seq_num 297) 2024-04-24T01:58:00.749 INFO:tasks.rados.rados.0.smithi079.stdout:612: finishing write tid 1 to smithi07929762-28 2024-04-24T01:58:00.778 INFO:tasks.rados.rados.0.smithi079.stdout:601: done (6 left) 2024-04-24T01:58:00.779 INFO:tasks.rados.rados.0.smithi079.stdout:602: done (5 left) 2024-04-24T01:58:00.779 INFO:tasks.rados.rados.0.smithi079.stdout:603: done (4 left) 2024-04-24T01:58:00.779 INFO:tasks.rados.rados.0.smithi079.stdout:607: done (3 left) 2024-04-24T01:58:00.779 INFO:tasks.rados.rados.0.smithi079.stdout:611: done (2 left) 2024-04-24T01:58:00.779 INFO:tasks.rados.rados.0.smithi079.stdout:613: done (1 left) 2024-04-24T01:58:00.779 INFO:tasks.rados.rados.0.smithi079.stdout:614: write oid 3 current snap is 78 2024-04-24T01:58:00.779 INFO:tasks.rados.rados.0.smithi079.stdout:614: seq_num 301 ranges {658907=787716,1930727=497772,3032909=25998} 2024-04-24T01:58:00.786 INFO:tasks.rados.rados.0.smithi079.stdout:614: writing smithi07929762-3 from 658907 to 1446623 tid 1 2024-04-24T01:58:00.792 INFO:tasks.rados.rados.0.smithi079.stdout:614: writing smithi07929762-3 from 1930727 to 2428499 tid 2 2024-04-24T01:58:00.795 INFO:tasks.rados.rados.0.smithi079.stdout:614: writing smithi07929762-3 from 3032909 to 3058907 tid 3 2024-04-24T01:58:00.796 INFO:tasks.rados.rados.0.smithi079.stdout:615: write oid 26 current snap is 78 2024-04-24T01:58:00.796 INFO:tasks.rados.rados.0.smithi079.stdout:612: finishing write tid 2 to smithi07929762-28 2024-04-24T01:58:00.796 INFO:tasks.rados.rados.0.smithi079.stdout:615: seq_num 302 ranges {675714=462965,1589209=641430,2943979=131735} 2024-04-24T01:58:00.805 INFO:tasks.rados.rados.0.smithi079.stdout:615: writing smithi07929762-26 from 675714 to 1138679 tid 1 2024-04-24T01:58:00.814 INFO:tasks.rados.rados.0.smithi079.stdout:615: writing smithi07929762-26 from 1589209 to 2230639 tid 2 2024-04-24T01:58:00.820 INFO:tasks.rados.rados.0.smithi079.stdout:615: writing smithi07929762-26 from 2943979 to 3075714 tid 3 2024-04-24T01:58:00.821 INFO:tasks.rados.rados.0.smithi079.stdout:616: delete oid 32 current snap is 78 2024-04-24T01:58:00.821 INFO:tasks.rados.rados.0.smithi079.stdout:612: finishing write tid 3 to smithi07929762-28 2024-04-24T01:58:00.821 INFO:tasks.rados.rados.0.smithi079.stdout:612: finishing write tid 4 to smithi07929762-28 2024-04-24T01:58:00.821 INFO:tasks.rados.rados.0.smithi079.stdout:612: finishing write tid 5 to smithi07929762-28 2024-04-24T01:58:00.821 INFO:tasks.rados.rados.0.smithi079.stdout:612: oid 28 updating version 0 to 173 2024-04-24T01:58:00.821 INFO:tasks.rados.rados.0.smithi079.stdout:612: oid 28 version 173 is already newer than 173 2024-04-24T01:58:00.821 INFO:tasks.rados.rados.0.smithi079.stdout:612: oid 28 version 173 is already newer than 173 2024-04-24T01:58:00.821 INFO:tasks.rados.rados.0.smithi079.stdout:612: oid 28 version 173 is already newer than 170 2024-04-24T01:58:00.821 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 28 v 173 (ObjNum 299 snap 78 seq_num 299) dirty exists 2024-04-24T01:58:00.821 INFO:tasks.rados.rados.0.smithi079.stdout:612: left oid 28 (ObjNum 299 snap 78 seq_num 299) 2024-04-24T01:58:00.837 INFO:tasks.rados.rados.0.smithi079.stdout:614: finishing write tid 1 to smithi07929762-3 2024-04-24T01:58:00.874 INFO:tasks.rados.rados.0.smithi079.stdout:614: finishing write tid 2 to smithi07929762-3 2024-04-24T01:58:00.877 INFO:tasks.rados.rados.0.smithi079.stdout:615: finishing write tid 1 to smithi07929762-26 2024-04-24T01:58:00.883 INFO:tasks.rados.rados.0.smithi079.stdout:614: finishing write tid 3 to smithi07929762-3 2024-04-24T01:58:00.883 INFO:tasks.rados.rados.0.smithi079.stdout:614: finishing write tid 4 to smithi07929762-3 2024-04-24T01:58:00.884 INFO:tasks.rados.rados.0.smithi079.stdout:614: finishing write tid 5 to smithi07929762-3 2024-04-24T01:58:00.884 INFO:tasks.rados.rados.0.smithi079.stdout:614: oid 3 updating version 0 to 271 2024-04-24T01:58:00.884 INFO:tasks.rados.rados.0.smithi079.stdout:614: oid 3 version 271 is already newer than 271 2024-04-24T01:58:00.884 INFO:tasks.rados.rados.0.smithi079.stdout:614: oid 3 version 271 is already newer than 271 2024-04-24T01:58:00.884 INFO:tasks.rados.rados.0.smithi079.stdout:614: oid 3 version 271 is already newer than 271 2024-04-24T01:58:00.884 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 3 v 271 (ObjNum 301 snap 78 seq_num 301) dirty exists 2024-04-24T01:58:00.884 INFO:tasks.rados.rados.0.smithi079.stdout:614: left oid 3 (ObjNum 301 snap 78 seq_num 301) 2024-04-24T01:58:00.953 INFO:tasks.rados.rados.0.smithi079.stdout:615: finishing write tid 2 to smithi07929762-26 2024-04-24T01:58:00.955 INFO:tasks.rados.rados.0.smithi079.stdout:615: finishing write tid 3 to smithi07929762-26 2024-04-24T01:58:00.958 INFO:tasks.rados.rados.0.smithi079.stdout:615: finishing write tid 4 to smithi07929762-26 2024-04-24T01:58:00.959 INFO:tasks.rados.rados.0.smithi079.stdout:612: done (3 left) 2024-04-24T01:58:00.959 INFO:tasks.rados.rados.0.smithi079.stdout:614: done (2 left) 2024-04-24T01:58:00.959 INFO:tasks.rados.rados.0.smithi079.stdout:616: done (1 left) 2024-04-24T01:58:00.959 INFO:tasks.rados.rados.0.smithi079.stdout:617: write oid 46 current snap is 78 2024-04-24T01:58:00.959 INFO:tasks.rados.rados.0.smithi079.stdout:617: seq_num 304 ranges {709328=613463,1989820=645099} 2024-04-24T01:58:00.966 INFO:tasks.rados.rados.0.smithi079.stdout:617: writing smithi07929762-46 from 709328 to 1322791 tid 1 2024-04-24T01:58:00.975 INFO:tasks.rados.rados.0.smithi079.stdout:617: writing smithi07929762-46 from 1989820 to 2634919 tid 2 2024-04-24T01:58:00.975 INFO:tasks.rados.rados.0.smithi079.stdout:618: read oid 34 snap -1 2024-04-24T01:58:00.975 INFO:tasks.rados.rados.0.smithi079.stdout:618: expect (ObjNum 244 snap 67 seq_num 244) 2024-04-24T01:58:00.975 INFO:tasks.rados.rados.0.smithi079.stdout:619: write (excl) oid 39 current snap is 78 2024-04-24T01:58:00.975 INFO:tasks.rados.rados.0.smithi079.stdout:615: finishing write tid 5 to smithi07929762-26 2024-04-24T01:58:00.975 INFO:tasks.rados.rados.0.smithi079.stdout:615: oid 26 updating version 0 to 215 2024-04-24T01:58:00.975 INFO:tasks.rados.rados.0.smithi079.stdout:615: oid 26 version 215 is already newer than 215 2024-04-24T01:58:00.975 INFO:tasks.rados.rados.0.smithi079.stdout:615: oid 26 version 215 is already newer than 215 2024-04-24T01:58:00.975 INFO:tasks.rados.rados.0.smithi079.stdout:615: oid 26 version 215 is already newer than 215 2024-04-24T01:58:00.975 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 26 v 215 (ObjNum 302 snap 78 seq_num 302) dirty exists 2024-04-24T01:58:00.975 INFO:tasks.rados.rados.0.smithi079.stdout:615: left oid 26 (ObjNum 302 snap 78 seq_num 302) 2024-04-24T01:58:00.975 INFO:tasks.rados.rados.0.smithi079.stdout:619: seq_num 305 ranges {726135=405065,1764655=788757,3026248=99887} 2024-04-24T01:58:00.980 INFO:tasks.rados.rados.0.smithi079.stdout:619: writing smithi07929762-39 from 726135 to 1131200 tid 1 2024-04-24T01:58:00.989 INFO:tasks.rados.rados.0.smithi079.stdout:619: writing smithi07929762-39 from 1764655 to 2553412 tid 2 2024-04-24T01:58:00.992 INFO:tasks.rados.rados.0.smithi079.stdout:619: writing smithi07929762-39 from 3026248 to 3126135 tid 3 2024-04-24T01:58:00.992 INFO:tasks.rados.rados.0.smithi079.stdout:615: done (3 left) 2024-04-24T01:58:00.992 INFO:tasks.rados.rados.0.smithi079.stdout:620: read oid 10 snap 72 2024-04-24T01:58:00.992 INFO:tasks.rados.rados.0.smithi079.stdout:620: expect (ObjNum 147 snap 20 seq_num 147) 2024-04-24T01:58:00.992 INFO:tasks.rados.rados.0.smithi079.stdout:621: delete oid 7 current snap is 78 2024-04-24T01:58:01.002 INFO:tasks.rados.rados.0.smithi079.stdout:617: finishing write tid 1 to smithi07929762-46 2024-04-24T01:58:01.021 INFO:tasks.rados.rados.0.smithi079.stdout:617: finishing write tid 2 to smithi07929762-46 2024-04-24T01:58:01.025 INFO:tasks.rados.rados.0.smithi079.stdout:619: finishing write tid 1 to smithi07929762-39 2024-04-24T01:58:01.032 INFO:tasks.rados.rados.0.smithi079.stdout:617: finishing write tid 3 to smithi07929762-46 2024-04-24T01:58:01.038 INFO:tasks.rados.rados.0.smithi079.stdout:617: finishing write tid 4 to smithi07929762-46 2024-04-24T01:58:01.038 INFO:tasks.rados.rados.0.smithi079.stdout:617: oid 46 updating version 0 to 145 2024-04-24T01:58:01.038 INFO:tasks.rados.rados.0.smithi079.stdout:617: oid 46 version 145 is already newer than 145 2024-04-24T01:58:01.038 INFO:tasks.rados.rados.0.smithi079.stdout:617: oid 46 version 145 is already newer than 145 2024-04-24T01:58:01.038 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 46 v 145 (ObjNum 304 snap 78 seq_num 304) dirty exists 2024-04-24T01:58:01.039 INFO:tasks.rados.rados.0.smithi079.stdout:617: left oid 46 (ObjNum 304 snap 78 seq_num 304) 2024-04-24T01:58:01.051 INFO:tasks.rados.rados.0.smithi079.stdout:619: finishing write tid 2 to smithi07929762-39 2024-04-24T01:58:01.052 INFO:tasks.rados.rados.0.smithi079.stdout:619: finishing write tid 3 to smithi07929762-39 2024-04-24T01:58:01.057 INFO:tasks.rados.rados.0.smithi079.stdout:620: expect (ObjNum 147 snap 20 seq_num 147) 2024-04-24T01:58:01.153 INFO:tasks.rados.rados.0.smithi079.stdout:618: expect (ObjNum 244 snap 67 seq_num 244) 2024-04-24T01:58:01.255 INFO:tasks.rados.rados.0.smithi079.stdout:619: finishing write tid 4 to smithi07929762-39 2024-04-24T01:58:01.255 INFO:tasks.rados.rados.0.smithi079.stdout:619: finishing write tid 5 to smithi07929762-39 2024-04-24T01:58:01.255 INFO:tasks.rados.rados.0.smithi079.stdout:619: oid 39 updating version 0 to 132 2024-04-24T01:58:01.255 INFO:tasks.rados.rados.0.smithi079.stdout:619: oid 39 updating version 132 to 135 2024-04-24T01:58:01.255 INFO:tasks.rados.rados.0.smithi079.stdout:619: oid 39 version 135 is already newer than 135 2024-04-24T01:58:01.255 INFO:tasks.rados.rados.0.smithi079.stdout:619: oid 39 version 135 is already newer than 135 2024-04-24T01:58:01.255 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 39 v 135 (ObjNum 305 snap 78 seq_num 305) dirty exists 2024-04-24T01:58:01.256 INFO:tasks.rados.rados.0.smithi079.stdout:619: left oid 39 (ObjNum 305 snap 78 seq_num 305) 2024-04-24T01:58:01.256 INFO:tasks.rados.rados.0.smithi079.stdout:617: done (4 left) 2024-04-24T01:58:01.256 INFO:tasks.rados.rados.0.smithi079.stdout:618: done (3 left) 2024-04-24T01:58:01.256 INFO:tasks.rados.rados.0.smithi079.stdout:619: done (2 left) 2024-04-24T01:58:01.256 INFO:tasks.rados.rados.0.smithi079.stdout:620: done (1 left) 2024-04-24T01:58:01.256 INFO:tasks.rados.rados.0.smithi079.stdout:621: done (0 left) 2024-04-24T01:58:01.256 INFO:tasks.rados.rados.0.smithi079.stdout:622: write oid 35 current snap is 78 2024-04-24T01:58:01.256 INFO:tasks.rados.rados.0.smithi079.stdout:622: seq_num 307 ranges {759749=555563,1765266=792426,2964741=195008} 2024-04-24T01:58:01.262 INFO:tasks.rados.rados.0.smithi079.stdout:622: writing smithi07929762-35 from 759749 to 1315312 tid 1 2024-04-24T01:58:01.268 INFO:tasks.rados.rados.0.smithi079.stdout:622: writing smithi07929762-35 from 1765266 to 2557692 tid 2 2024-04-24T01:58:01.273 INFO:tasks.rados.rados.0.smithi079.stdout:622: writing smithi07929762-35 from 2964741 to 3159749 tid 3 2024-04-24T01:58:01.273 INFO:tasks.rados.rados.0.smithi079.stdout:623: read oid 40 snap -1 2024-04-24T01:58:01.273 INFO:tasks.rados.rados.0.smithi079.stdout:623: expect deleted 2024-04-24T01:58:01.274 INFO:tasks.rados.rados.0.smithi079.stdout:624: snap_create 2024-04-24T01:58:01.319 INFO:tasks.rados.rados.0.smithi079.stdout:622: finishing write tid 1 to smithi07929762-35 2024-04-24T01:58:01.320 INFO:tasks.rados.rados.0.smithi079.stdout:622: finishing write tid 2 to smithi07929762-35 2024-04-24T01:58:01.323 INFO:tasks.rados.rados.0.smithi079.stdout:622: finishing write tid 3 to smithi07929762-35 2024-04-24T01:58:01.327 INFO:tasks.rados.rados.0.smithi079.stdout:622: finishing write tid 4 to smithi07929762-35 2024-04-24T01:58:01.329 INFO:tasks.rados.rados.0.smithi079.stdout:622: finishing write tid 5 to smithi07929762-35 2024-04-24T01:58:01.329 INFO:tasks.rados.rados.0.smithi079.stdout:622: oid 35 updating version 0 to 169 2024-04-24T01:58:01.329 INFO:tasks.rados.rados.0.smithi079.stdout:622: oid 35 version 169 is already newer than 169 2024-04-24T01:58:01.329 INFO:tasks.rados.rados.0.smithi079.stdout:622: oid 35 version 169 is already newer than 169 2024-04-24T01:58:01.329 INFO:tasks.rados.rados.0.smithi079.stdout:622: oid 35 version 169 is already newer than 169 2024-04-24T01:58:01.329 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 35 v 169 (ObjNum 307 snap 78 seq_num 307) dirty exists 2024-04-24T01:58:01.329 INFO:tasks.rados.rados.0.smithi079.stdout:622: left oid 35 (ObjNum 307 snap 78 seq_num 307) 2024-04-24T01:58:01.903 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T01:58:02.009 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:58:02.007+0000 7f14c7d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:02.013 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:58:02.012+0000 7f14c7d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:02.080 INFO:tasks.rados.rados.0.smithi079.stdout:622: done (2 left) 2024-04-24T01:58:02.080 INFO:tasks.rados.rados.0.smithi079.stdout:623: done (1 left) 2024-04-24T01:58:02.081 INFO:tasks.rados.rados.0.smithi079.stdout:624: done (0 left) 2024-04-24T01:58:02.081 INFO:tasks.rados.rados.0.smithi079.stdout:625: delete oid 4 current snap is 79 2024-04-24T01:58:02.133 INFO:tasks.rados.rados.0.smithi079.stdout:625: done (0 left) 2024-04-24T01:58:02.133 INFO:tasks.rados.rados.0.smithi079.stdout:626: delete oid 1 current snap is 79 2024-04-24T01:58:02.163 INFO:tasks.rados.rados.0.smithi079.stdout:626: done (0 left) 2024-04-24T01:58:02.163 INFO:tasks.rados.rados.0.smithi079.stdout:627: read oid 41 snap 53 2024-04-24T01:58:02.163 INFO:tasks.rados.rados.0.smithi079.stdout:627: expect (ObjNum 113 snap 10 seq_num 113) 2024-04-24T01:58:02.164 INFO:tasks.rados.rados.0.smithi079.stdout:628: read oid 28 snap -1 2024-04-24T01:58:02.164 INFO:tasks.rados.rados.0.smithi079.stdout:628: expect (ObjNum 299 snap 78 seq_num 299) 2024-04-24T01:58:02.164 INFO:tasks.rados.rados.0.smithi079.stdout:629: snap_remove snap 63 2024-04-24T01:58:03.084 INFO:tasks.rados.rados.0.smithi079.stdout:629: done (2 left) 2024-04-24T01:58:03.085 INFO:tasks.rados.rados.0.smithi079.stdout:630: delete oid 4 current snap is 79 2024-04-24T01:58:03.085 INFO:tasks.rados.rados.0.smithi079.stdout:627: expect (ObjNum 113 snap 10 seq_num 113) 2024-04-24T01:58:03.304 INFO:tasks.rados.rados.0.smithi079.stdout:628: expect (ObjNum 299 snap 78 seq_num 299) 2024-04-24T01:58:03.568 INFO:tasks.rados.rados.0.smithi079.stdout:627: done (2 left) 2024-04-24T01:58:03.568 INFO:tasks.rados.rados.0.smithi079.stdout:628: done (1 left) 2024-04-24T01:58:03.568 INFO:tasks.rados.rados.0.smithi079.stdout:630: done (0 left) 2024-04-24T01:58:03.568 INFO:tasks.rados.rados.0.smithi079.stdout:631: write oid 44 current snap is 79 2024-04-24T01:58:03.568 INFO:tasks.rados.rados.0.smithi079.stdout:631: seq_num 311 ranges {426977=456559,1482841=799764,2958080=268897} 2024-04-24T01:58:03.572 INFO:tasks.rados.rados.0.smithi079.stdout:631: writing smithi07929762-44 from 426977 to 883536 tid 1 2024-04-24T01:58:03.580 INFO:tasks.rados.rados.0.smithi079.stdout:631: writing smithi07929762-44 from 1482841 to 2282605 tid 2 2024-04-24T01:58:03.585 INFO:tasks.rados.rados.0.smithi079.stdout:631: writing smithi07929762-44 from 2958080 to 3226977 tid 3 2024-04-24T01:58:03.585 INFO:tasks.rados.rados.0.smithi079.stdout:632: delete oid 19 current snap is 79 2024-04-24T01:58:03.620 INFO:tasks.rados.rados.0.smithi079.stdout:632: done (1 left) 2024-04-24T01:58:03.620 INFO:tasks.rados.rados.0.smithi079.stdout:633: read oid 42 snap -1 2024-04-24T01:58:03.620 INFO:tasks.rados.rados.0.smithi079.stdout:633: expect deleted 2024-04-24T01:58:03.620 INFO:tasks.rados.rados.0.smithi079.stdout:634: read oid 13 snap -1 2024-04-24T01:58:03.620 INFO:tasks.rados.rados.0.smithi079.stdout:634: expect deleted 2024-04-24T01:58:03.620 INFO:tasks.rados.rados.0.smithi079.stdout:635: read oid 15 snap -1 2024-04-24T01:58:03.620 INFO:tasks.rados.rados.0.smithi079.stdout:635: expect (ObjNum 197 snap 40 seq_num 197) 2024-04-24T01:58:03.620 INFO:tasks.rados.rados.0.smithi079.stdout:636: read oid 22 snap -1 2024-04-24T01:58:03.621 INFO:tasks.rados.rados.0.smithi079.stdout:636: expect (ObjNum 277 snap 72 seq_num 277) 2024-04-24T01:58:03.621 INFO:tasks.rados.rados.0.smithi079.stdout:637: write (excl) oid 32 current snap is 79 2024-04-24T01:58:03.621 INFO:tasks.rados.rados.0.smithi079.stdout:637: seq_num 313 ranges {460591=723410,1716158=403433,2729279=440140} 2024-04-24T01:58:03.629 INFO:tasks.rados.rados.0.smithi079.stdout:637: writing smithi07929762-32 from 460591 to 1184001 tid 1 2024-04-24T01:58:03.636 INFO:tasks.rados.rados.0.smithi079.stdout:637: writing smithi07929762-32 from 1716158 to 2119591 tid 2 2024-04-24T01:58:03.644 INFO:tasks.rados.rados.0.smithi079.stdout:637: writing smithi07929762-32 from 2729279 to 3169419 tid 3 2024-04-24T01:58:03.644 INFO:tasks.rados.rados.0.smithi079.stdout:638: delete oid 31 current snap is 79 2024-04-24T01:58:03.645 INFO:tasks.rados.rados.0.smithi079.stdout:636: expect (ObjNum 277 snap 72 seq_num 277) 2024-04-24T01:58:03.831 INFO:tasks.rados.rados.0.smithi079.stdout:635: expect (ObjNum 197 snap 40 seq_num 197) 2024-04-24T01:58:03.955 INFO:tasks.rados.rados.0.smithi079.stdout:637: finishing write tid 1 to smithi07929762-32 2024-04-24T01:58:03.955 INFO:tasks.rados.rados.0.smithi079.stdout:631: finishing write tid 1 to smithi07929762-44 2024-04-24T01:58:03.955 INFO:tasks.rados.rados.0.smithi079.stdout:631: finishing write tid 2 to smithi07929762-44 2024-04-24T01:58:03.955 INFO:tasks.rados.rados.0.smithi079.stdout:631: finishing write tid 3 to smithi07929762-44 2024-04-24T01:58:03.955 INFO:tasks.rados.rados.0.smithi079.stdout:631: finishing write tid 4 to smithi07929762-44 2024-04-24T01:58:03.955 INFO:tasks.rados.rados.0.smithi079.stdout:637: finishing write tid 2 to smithi07929762-32 2024-04-24T01:58:03.955 INFO:tasks.rados.rados.0.smithi079.stdout:637: finishing write tid 3 to smithi07929762-32 2024-04-24T01:58:03.955 INFO:tasks.rados.rados.0.smithi079.stdout:631: finishing write tid 5 to smithi07929762-44 2024-04-24T01:58:03.955 INFO:tasks.rados.rados.0.smithi079.stdout:631: oid 44 updating version 0 to 144 2024-04-24T01:58:03.955 INFO:tasks.rados.rados.0.smithi079.stdout:631: oid 44 version 144 is already newer than 144 2024-04-24T01:58:03.956 INFO:tasks.rados.rados.0.smithi079.stdout:631: oid 44 version 144 is already newer than 144 2024-04-24T01:58:03.956 INFO:tasks.rados.rados.0.smithi079.stdout:631: oid 44 version 144 is already newer than 144 2024-04-24T01:58:03.956 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 44 v 144 (ObjNum 311 snap 79 seq_num 311) dirty exists 2024-04-24T01:58:03.956 INFO:tasks.rados.rados.0.smithi079.stdout:631: left oid 44 (ObjNum 311 snap 79 seq_num 311) 2024-04-24T01:58:03.956 INFO:tasks.rados.rados.0.smithi079.stdout:637: finishing write tid 4 to smithi07929762-32 2024-04-24T01:58:03.956 INFO:tasks.rados.rados.0.smithi079.stdout:637: finishing write tid 5 to smithi07929762-32 2024-04-24T01:58:03.956 INFO:tasks.rados.rados.0.smithi079.stdout:637: oid 32 updating version 0 to 106 2024-04-24T01:58:03.956 INFO:tasks.rados.rados.0.smithi079.stdout:637: oid 32 updating version 106 to 109 2024-04-24T01:58:03.956 INFO:tasks.rados.rados.0.smithi079.stdout:637: oid 32 version 109 is already newer than 109 2024-04-24T01:58:03.956 INFO:tasks.rados.rados.0.smithi079.stdout:637: oid 32 version 109 is already newer than 109 2024-04-24T01:58:03.956 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 32 v 109 (ObjNum 313 snap 79 seq_num 313) dirty exists 2024-04-24T01:58:03.956 INFO:tasks.rados.rados.0.smithi079.stdout:637: left oid 32 (ObjNum 313 snap 79 seq_num 313) 2024-04-24T01:58:03.956 INFO:tasks.rados.rados.0.smithi079.stdout:631: done (6 left) 2024-04-24T01:58:03.957 INFO:tasks.rados.rados.0.smithi079.stdout:633: done (5 left) 2024-04-24T01:58:03.957 INFO:tasks.rados.rados.0.smithi079.stdout:634: done (4 left) 2024-04-24T01:58:03.957 INFO:tasks.rados.rados.0.smithi079.stdout:635: done (3 left) 2024-04-24T01:58:03.957 INFO:tasks.rados.rados.0.smithi079.stdout:636: done (2 left) 2024-04-24T01:58:03.957 INFO:tasks.rados.rados.0.smithi079.stdout:637: done (1 left) 2024-04-24T01:58:03.957 INFO:tasks.rados.rados.0.smithi079.stdout:638: done (0 left) 2024-04-24T01:58:03.957 INFO:tasks.rados.rados.0.smithi079.stdout:639: read oid 3 snap 76 2024-04-24T01:58:03.957 INFO:tasks.rados.rados.0.smithi079.stdout:639: expect (ObjNum 151 snap 20 seq_num 151) 2024-04-24T01:58:03.957 INFO:tasks.rados.rados.0.smithi079.stdout:640: delete oid 49 current snap is 79 2024-04-24T01:58:03.999 INFO:tasks.rados.rados.0.smithi079.stdout:640: done (1 left) 2024-04-24T01:58:03.999 INFO:tasks.rados.rados.0.smithi079.stdout:641: snap_remove snap 72 2024-04-24T01:58:04.060 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-04-24T01:58:04.060 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-04-24T01:58:04.060 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-04-24T01:58:04.060 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-04-24T01:58:04.086 INFO:tasks.rados.rados.0.smithi079.stdout:641: done (1 left) 2024-04-24T01:58:04.087 INFO:tasks.rados.rados.0.smithi079.stdout:642: snap_remove snap 77 2024-04-24T01:58:04.113 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T01:58:04.177 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:58:04.170+0000 7f3fbf91d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:04.181 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:58:04.180+0000 7f3fbf91d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:04.202 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T01:58:04.202 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T01:58:04.202 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T01:58:04.202 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T01:58:04.202 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T01:58:04.202 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T01:58:04.202 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T01:58:04.202 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T01:58:04.202 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T01:58:04.202 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T01:58:04.203 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T01:58:04.203 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T01:58:04.206 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T01:58:04.206 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T01:58:04.207 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-04-24T01:58:04.243 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:58:04.240+0000 7f5023c64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:04.248 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:58:04.247+0000 7f5023c64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:04.338 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:58:04.332+0000 7fb06dc24640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:04.343 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:58:04.343+0000 7fb06dc24640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:04.360 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T01:58:04.360 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T01:58:04.360 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T01:58:04.360 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T01:58:04.360 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T01:58:04.361 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T01:58:04.361 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T01:58:04.361 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T01:58:04.361 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T01:58:04.361 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T01:58:04.361 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T01:58:04.361 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T01:58:04.363 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T01:58:04.364 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T01:58:04.364 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-04-24T01:58:04.536 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:58:04.530+0000 7f700b02d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:04.540 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:58:04.539+0000 7f700b02d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:04.556 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T01:58:04.556 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T01:58:04.556 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T01:58:04.556 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T01:58:04.556 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T01:58:04.556 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T01:58:04.556 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T01:58:04.556 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T01:58:04.556 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T01:58:04.556 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T01:58:04.556 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T01:58:04.557 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T01:58:04.559 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T01:58:04.559 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T01:58:04.559 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-04-24T01:58:04.666 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:58:04.661+0000 7face4f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:04.670 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:58:04.669+0000 7face4f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:04.686 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T01:58:04.686 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T01:58:04.686 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T01:58:04.686 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T01:58:04.686 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T01:58:04.686 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T01:58:04.686 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T01:58:04.686 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T01:58:04.686 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T01:58:04.686 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T01:58:04.687 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T01:58:04.687 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T01:58:04.689 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T01:58:04.689 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T01:58:05.091 INFO:tasks.rados.rados.0.smithi079.stdout:642: done (1 left) 2024-04-24T01:58:05.091 INFO:tasks.rados.rados.0.smithi079.stdout:643: read oid 5 snap -1 2024-04-24T01:58:05.091 INFO:tasks.rados.rados.0.smithi079.stdout:643: expect (ObjNum 291 snap 76 seq_num 291) 2024-04-24T01:58:05.091 INFO:tasks.rados.rados.0.smithi079.stdout:639: expect (ObjNum 151 snap 20 seq_num 151) 2024-04-24T01:58:05.205 INFO:tasks.rados.rados.0.smithi079.stdout:643: expect (ObjNum 291 snap 76 seq_num 291) 2024-04-24T01:58:05.350 INFO:tasks.rados.rados.0.smithi079.stdout:639: done (1 left) 2024-04-24T01:58:05.350 INFO:tasks.rados.rados.0.smithi079.stdout:643: done (0 left) 2024-04-24T01:58:05.351 INFO:tasks.rados.rados.0.smithi079.stdout:644: write oid 40 current snap is 79 2024-04-24T01:58:05.351 INFO:tasks.rados.rados.0.smithi079.stdout:644: seq_num 316 ranges {511012=549157,1775251=550760,3095195=215817} 2024-04-24T01:58:05.356 INFO:tasks.rados.rados.0.smithi079.stdout:644: writing smithi07929762-40 from 511012 to 1060169 tid 1 2024-04-24T01:58:05.363 INFO:tasks.rados.rados.0.smithi079.stdout:644: writing smithi07929762-40 from 1775251 to 2326011 tid 2 2024-04-24T01:58:05.369 INFO:tasks.rados.rados.0.smithi079.stdout:644: writing smithi07929762-40 from 3095195 to 3311012 tid 3 2024-04-24T01:58:05.369 INFO:tasks.rados.rados.0.smithi079.stdout:645: delete oid 18 current snap is 79 2024-04-24T01:58:05.416 INFO:tasks.rados.rados.0.smithi079.stdout:645: done (1 left) 2024-04-24T01:58:05.416 INFO:tasks.rados.rados.0.smithi079.stdout:646: snap_remove snap 78 2024-04-24T01:58:06.095 INFO:tasks.rados.rados.0.smithi079.stdout:646: done (1 left) 2024-04-24T01:58:06.096 INFO:tasks.rados.rados.0.smithi079.stdout:647: write oid 34 current snap is 79 2024-04-24T01:58:06.096 INFO:tasks.rados.rados.0.smithi079.stdout:644: finishing write tid 1 to smithi07929762-40 2024-04-24T01:58:06.096 INFO:tasks.rados.rados.0.smithi079.stdout:647: seq_num 318 ranges {544626=699655,1892215=554429,3150041=194585} 2024-04-24T01:58:06.102 INFO:tasks.rados.rados.0.smithi079.stdout:647: writing smithi07929762-34 from 544626 to 1244281 tid 1 2024-04-24T01:58:06.109 INFO:tasks.rados.rados.0.smithi079.stdout:647: writing smithi07929762-34 from 1892215 to 2446644 tid 2 2024-04-24T01:58:06.114 INFO:tasks.rados.rados.0.smithi079.stdout:647: writing smithi07929762-34 from 3150041 to 3344626 tid 3 2024-04-24T01:58:06.114 INFO:tasks.rados.rados.0.smithi079.stdout:648: delete oid 33 current snap is 79 2024-04-24T01:58:06.114 INFO:tasks.rados.rados.0.smithi079.stdout:644: finishing write tid 2 to smithi07929762-40 2024-04-24T01:58:06.114 INFO:tasks.rados.rados.0.smithi079.stdout:644: finishing write tid 3 to smithi07929762-40 2024-04-24T01:58:06.115 INFO:tasks.rados.rados.0.smithi079.stdout:644: finishing write tid 4 to smithi07929762-40 2024-04-24T01:58:06.115 INFO:tasks.rados.rados.0.smithi079.stdout:644: finishing write tid 5 to smithi07929762-40 2024-04-24T01:58:06.115 INFO:tasks.rados.rados.0.smithi079.stdout:644: oid 40 updating version 0 to 224 2024-04-24T01:58:06.115 INFO:tasks.rados.rados.0.smithi079.stdout:644: oid 40 version 224 is already newer than 224 2024-04-24T01:58:06.115 INFO:tasks.rados.rados.0.smithi079.stdout:644: oid 40 version 224 is already newer than 224 2024-04-24T01:58:06.115 INFO:tasks.rados.rados.0.smithi079.stdout:644: oid 40 version 224 is already newer than 224 2024-04-24T01:58:06.115 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 40 v 224 (ObjNum 316 snap 79 seq_num 316) dirty exists 2024-04-24T01:58:06.115 INFO:tasks.rados.rados.0.smithi079.stdout:644: left oid 40 (ObjNum 316 snap 79 seq_num 316) 2024-04-24T01:58:06.158 INFO:tasks.rados.rados.0.smithi079.stdout:644: done (2 left) 2024-04-24T01:58:06.158 INFO:tasks.rados.rados.0.smithi079.stdout:648: done (1 left) 2024-04-24T01:58:06.158 INFO:tasks.rados.rados.0.smithi079.stdout:649: read oid 17 snap -1 2024-04-24T01:58:06.159 INFO:tasks.rados.rados.0.smithi079.stdout:649: expect (ObjNum 269 snap 70 seq_num 269) 2024-04-24T01:58:06.159 INFO:tasks.rados.rados.0.smithi079.stdout:650: delete oid 13 current snap is 79 2024-04-24T01:58:06.162 INFO:tasks.rados.rados.0.smithi079.stdout:650: done (2 left) 2024-04-24T01:58:06.162 INFO:tasks.rados.rados.0.smithi079.stdout:651: read oid 12 snap -1 2024-04-24T01:58:06.162 INFO:tasks.rados.rados.0.smithi079.stdout:651: expect deleted 2024-04-24T01:58:06.162 INFO:tasks.rados.rados.0.smithi079.stdout:652: delete oid 13 current snap is 79 2024-04-24T01:58:06.166 INFO:tasks.rados.rados.0.smithi079.stdout:652: done (3 left) 2024-04-24T01:58:06.166 INFO:tasks.rados.rados.0.smithi079.stdout:653: read oid 24 snap -1 2024-04-24T01:58:06.166 INFO:tasks.rados.rados.0.smithi079.stdout:653: expect (ObjNum 297 snap 78 seq_num 297) 2024-04-24T01:58:06.167 INFO:tasks.rados.rados.0.smithi079.stdout:654: snap_remove snap 67 2024-04-24T01:58:06.391 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T01:58:06.496 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:58:06.494+0000 7fe3b5e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:06.500 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:58:06.499+0000 7fe3b5e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:07.098 INFO:tasks.rados.rados.0.smithi079.stdout:654: done (4 left) 2024-04-24T01:58:07.098 INFO:tasks.rados.rados.0.smithi079.stdout:655: read oid 40 snap -1 2024-04-24T01:58:07.099 INFO:tasks.rados.rados.0.smithi079.stdout:655: expect (ObjNum 316 snap 79 seq_num 316) 2024-04-24T01:58:07.099 INFO:tasks.rados.rados.0.smithi079.stdout:656: write oid 11 current snap is 79 2024-04-24T01:58:07.099 INFO:tasks.rados.rados.0.smithi079.stdout:656: seq_num 322 ranges {611854=717004,2126143=561767,3259733=152121} 2024-04-24T01:58:07.108 INFO:tasks.rados.rados.0.smithi079.stdout:656: writing smithi07929762-11 from 611854 to 1328858 tid 1 2024-04-24T01:58:07.118 INFO:tasks.rados.rados.0.smithi079.stdout:656: writing smithi07929762-11 from 2126143 to 2687910 tid 2 2024-04-24T01:58:07.124 INFO:tasks.rados.rados.0.smithi079.stdout:656: writing smithi07929762-11 from 3259733 to 3411854 tid 3 2024-04-24T01:58:07.124 INFO:tasks.rados.rados.0.smithi079.stdout:657: read oid 39 snap -1 2024-04-24T01:58:07.124 INFO:tasks.rados.rados.0.smithi079.stdout:657: expect (ObjNum 305 snap 78 seq_num 305) 2024-04-24T01:58:07.124 INFO:tasks.rados.rados.0.smithi079.stdout:658: read oid 49 snap -1 2024-04-24T01:58:07.124 INFO:tasks.rados.rados.0.smithi079.stdout:658: expect deleted 2024-04-24T01:58:07.124 INFO:tasks.rados.rados.0.smithi079.stdout:659: write (excl) oid 47 current snap is 79 2024-04-24T01:58:07.124 INFO:tasks.rados.rados.0.smithi079.stdout:659: seq_num 323 ranges {628661=792253,2184625=421778,3403509=25152} 2024-04-24T01:58:07.135 INFO:tasks.rados.rados.0.smithi079.stdout:659: writing smithi07929762-47 from 628661 to 1420914 tid 1 2024-04-24T01:58:07.145 INFO:tasks.rados.rados.0.smithi079.stdout:659: writing smithi07929762-47 from 2184625 to 2606403 tid 2 2024-04-24T01:58:07.149 INFO:tasks.rados.rados.0.smithi079.stdout:659: writing smithi07929762-47 from 3403509 to 3428661 tid 3 2024-04-24T01:58:07.149 INFO:tasks.rados.rados.0.smithi079.stdout:660: read oid 14 snap -1 2024-04-24T01:58:07.149 INFO:tasks.rados.rados.0.smithi079.stdout:660: expect (ObjNum 219 snap 49 seq_num 219) 2024-04-24T01:58:07.149 INFO:tasks.rados.rados.0.smithi079.stdout:661: write (excl) oid 25 current snap is 79 2024-04-24T01:58:07.149 INFO:tasks.rados.rados.0.smithi079.stdout:661: seq_num 324 ranges {645468=467502,1726754=565436,2798226=596838} 2024-04-24T01:58:07.157 INFO:tasks.rados.rados.0.smithi079.stdout:661: writing smithi07929762-25 from 645468 to 1112970 tid 1 2024-04-24T01:58:07.166 INFO:tasks.rados.rados.0.smithi079.stdout:661: writing smithi07929762-25 from 1726754 to 2292190 tid 2 2024-04-24T01:58:07.176 INFO:tasks.rados.rados.0.smithi079.stdout:661: writing smithi07929762-25 from 2798226 to 3395064 tid 3 2024-04-24T01:58:07.176 INFO:tasks.rados.rados.0.smithi079.stdout:662: snap_remove snap 76 2024-04-24T01:58:08.101 INFO:tasks.rados.rados.0.smithi079.stdout:662: done (11 left) 2024-04-24T01:58:08.101 INFO:tasks.rados.rados.0.smithi079.stdout:663: delete oid 27 current snap is 79 2024-04-24T01:58:08.101 INFO:tasks.rados.rados.0.smithi079.stdout:647: finishing write tid 1 to smithi07929762-34 2024-04-24T01:58:08.102 INFO:tasks.rados.rados.0.smithi079.stdout:649: expect (ObjNum 269 snap 70 seq_num 269) 2024-04-24T01:58:08.347 INFO:tasks.rados.rados.0.smithi079.stdout:647: finishing write tid 2 to smithi07929762-34 2024-04-24T01:58:08.347 INFO:tasks.rados.rados.0.smithi079.stdout:647: finishing write tid 3 to smithi07929762-34 2024-04-24T01:58:08.347 INFO:tasks.rados.rados.0.smithi079.stdout:647: finishing write tid 4 to smithi07929762-34 2024-04-24T01:58:08.347 INFO:tasks.rados.rados.0.smithi079.stdout:647: finishing write tid 5 to smithi07929762-34 2024-04-24T01:58:08.347 INFO:tasks.rados.rados.0.smithi079.stdout:647: oid 34 updating version 0 to 156 2024-04-24T01:58:08.347 INFO:tasks.rados.rados.0.smithi079.stdout:647: oid 34 version 156 is already newer than 156 2024-04-24T01:58:08.348 INFO:tasks.rados.rados.0.smithi079.stdout:647: oid 34 version 156 is already newer than 156 2024-04-24T01:58:08.348 INFO:tasks.rados.rados.0.smithi079.stdout:647: oid 34 version 156 is already newer than 156 2024-04-24T01:58:08.348 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 34 v 156 (ObjNum 318 snap 79 seq_num 318) dirty exists 2024-04-24T01:58:08.348 INFO:tasks.rados.rados.0.smithi079.stdout:647: left oid 34 (ObjNum 318 snap 79 seq_num 318) 2024-04-24T01:58:08.348 INFO:tasks.rados.rados.0.smithi079.stdout:647: done (11 left) 2024-04-24T01:58:08.348 INFO:tasks.rados.rados.0.smithi079.stdout:649: done (10 left) 2024-04-24T01:58:08.348 INFO:tasks.rados.rados.0.smithi079.stdout:651: done (9 left) 2024-04-24T01:58:08.348 INFO:tasks.rados.rados.0.smithi079.stdout:663: done (8 left) 2024-04-24T01:58:08.348 INFO:tasks.rados.rados.0.smithi079.stdout:664: delete oid 17 current snap is 79 2024-04-24T01:58:08.348 INFO:tasks.rados.rados.0.smithi079.stdout:653: expect (ObjNum 297 snap 78 seq_num 297) 2024-04-24T01:58:08.592 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T01:58:08.665 INFO:tasks.rados.rados.0.smithi079.stdout:653: done (8 left) 2024-04-24T01:58:08.665 INFO:tasks.rados.rados.0.smithi079.stdout:664: done (7 left) 2024-04-24T01:58:08.665 INFO:tasks.rados.rados.0.smithi079.stdout:665: read oid 9 snap -1 2024-04-24T01:58:08.666 INFO:tasks.rados.rados.0.smithi079.stdout:665: expect (ObjNum 243 snap 67 seq_num 243) 2024-04-24T01:58:08.666 INFO:tasks.rados.rados.0.smithi079.stdout:666: read oid 13 snap -1 2024-04-24T01:58:08.666 INFO:tasks.rados.rados.0.smithi079.stdout:666: expect deleted 2024-04-24T01:58:08.666 INFO:tasks.rados.rados.0.smithi079.stdout:667: read oid 17 snap -1 2024-04-24T01:58:08.666 INFO:tasks.rados.rados.0.smithi079.stdout:667: expect deleted 2024-04-24T01:58:08.666 INFO:tasks.rados.rados.0.smithi079.stdout:668: delete oid 3 current snap is 79 2024-04-24T01:58:08.666 INFO:tasks.rados.rados.0.smithi079.stdout:655: expect (ObjNum 316 snap 79 seq_num 316) 2024-04-24T01:58:08.714 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:58:08.711+0000 7f081f31d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:08.718 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:58:08.717+0000 7f081f31d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:08.816 INFO:tasks.rados.rados.0.smithi079.stdout:657: expect (ObjNum 305 snap 78 seq_num 305) 2024-04-24T01:58:09.007 INFO:tasks.rados.rados.0.smithi079.stdout:656: finishing write tid 1 to smithi07929762-11 2024-04-24T01:58:09.007 INFO:tasks.rados.rados.0.smithi079.stdout:659: finishing write tid 1 to smithi07929762-47 2024-04-24T01:58:09.007 INFO:tasks.rados.rados.0.smithi079.stdout:656: finishing write tid 2 to smithi07929762-11 2024-04-24T01:58:09.007 INFO:tasks.rados.rados.0.smithi079.stdout:656: finishing write tid 3 to smithi07929762-11 2024-04-24T01:58:09.007 INFO:tasks.rados.rados.0.smithi079.stdout:656: finishing write tid 4 to smithi07929762-11 2024-04-24T01:58:09.007 INFO:tasks.rados.rados.0.smithi079.stdout:659: finishing write tid 2 to smithi07929762-47 2024-04-24T01:58:09.007 INFO:tasks.rados.rados.0.smithi079.stdout:656: finishing write tid 5 to smithi07929762-11 2024-04-24T01:58:09.007 INFO:tasks.rados.rados.0.smithi079.stdout:656: oid 11 updating version 0 to 164 2024-04-24T01:58:09.007 INFO:tasks.rados.rados.0.smithi079.stdout:656: oid 11 version 164 is already newer than 164 2024-04-24T01:58:09.007 INFO:tasks.rados.rados.0.smithi079.stdout:656: oid 11 version 164 is already newer than 164 2024-04-24T01:58:09.007 INFO:tasks.rados.rados.0.smithi079.stdout:656: oid 11 version 164 is already newer than 164 2024-04-24T01:58:09.007 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 11 v 164 (ObjNum 322 snap 79 seq_num 322) dirty exists 2024-04-24T01:58:09.008 INFO:tasks.rados.rados.0.smithi079.stdout:656: left oid 11 (ObjNum 322 snap 79 seq_num 322) 2024-04-24T01:58:09.008 INFO:tasks.rados.rados.0.smithi079.stdout:659: finishing write tid 3 to smithi07929762-47 2024-04-24T01:58:09.008 INFO:tasks.rados.rados.0.smithi079.stdout:655: done (10 left) 2024-04-24T01:58:09.008 INFO:tasks.rados.rados.0.smithi079.stdout:656: done (9 left) 2024-04-24T01:58:09.008 INFO:tasks.rados.rados.0.smithi079.stdout:657: done (8 left) 2024-04-24T01:58:09.008 INFO:tasks.rados.rados.0.smithi079.stdout:658: done (7 left) 2024-04-24T01:58:09.008 INFO:tasks.rados.rados.0.smithi079.stdout:668: done (6 left) 2024-04-24T01:58:09.008 INFO:tasks.rados.rados.0.smithi079.stdout:669: read oid 4 snap 71 2024-04-24T01:58:09.008 INFO:tasks.rados.rados.0.smithi079.stdout:669: expect deleted 2024-04-24T01:58:09.008 INFO:tasks.rados.rados.0.smithi079.stdout:670: delete oid 40 current snap is 79 2024-04-24T01:58:09.008 INFO:tasks.rados.rados.0.smithi079.stdout:659: finishing write tid 4 to smithi07929762-47 2024-04-24T01:58:09.008 INFO:tasks.rados.rados.0.smithi079.stdout:659: finishing write tid 5 to smithi07929762-47 2024-04-24T01:58:09.008 INFO:tasks.rados.rados.0.smithi079.stdout:659: oid 47 updating version 0 to 232 2024-04-24T01:58:09.008 INFO:tasks.rados.rados.0.smithi079.stdout:659: oid 47 version 232 is already newer than 232 2024-04-24T01:58:09.008 INFO:tasks.rados.rados.0.smithi079.stdout:659: oid 47 version 232 is already newer than 232 2024-04-24T01:58:09.008 INFO:tasks.rados.rados.0.smithi079.stdout:659: oid 47 version 232 is already newer than 229 2024-04-24T01:58:09.009 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 47 v 232 (ObjNum 323 snap 79 seq_num 323) dirty exists 2024-04-24T01:58:09.009 INFO:tasks.rados.rados.0.smithi079.stdout:659: left oid 47 (ObjNum 323 snap 79 seq_num 323) 2024-04-24T01:58:09.009 INFO:tasks.rados.rados.0.smithi079.stdout:661: finishing write tid 1 to smithi07929762-25 2024-04-24T01:58:09.009 INFO:tasks.rados.rados.0.smithi079.stdout:660: expect (ObjNum 219 snap 49 seq_num 219) 2024-04-24T01:58:09.310 INFO:tasks.rados.rados.0.smithi079.stdout:661: finishing write tid 2 to smithi07929762-25 2024-04-24T01:58:09.310 INFO:tasks.rados.rados.0.smithi079.stdout:661: finishing write tid 3 to smithi07929762-25 2024-04-24T01:58:09.310 INFO:tasks.rados.rados.0.smithi079.stdout:661: finishing write tid 4 to smithi07929762-25 2024-04-24T01:58:09.310 INFO:tasks.rados.rados.0.smithi079.stdout:661: finishing write tid 5 to smithi07929762-25 2024-04-24T01:58:09.310 INFO:tasks.rados.rados.0.smithi079.stdout:661: oid 25 updating version 0 to 234 2024-04-24T01:58:09.310 INFO:tasks.rados.rados.0.smithi079.stdout:661: oid 25 updating version 234 to 237 2024-04-24T01:58:09.311 INFO:tasks.rados.rados.0.smithi079.stdout:661: oid 25 version 237 is already newer than 237 2024-04-24T01:58:09.311 INFO:tasks.rados.rados.0.smithi079.stdout:661: oid 25 version 237 is already newer than 237 2024-04-24T01:58:09.311 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 25 v 237 (ObjNum 324 snap 79 seq_num 324) dirty exists 2024-04-24T01:58:09.311 INFO:tasks.rados.rados.0.smithi079.stdout:661: left oid 25 (ObjNum 324 snap 79 seq_num 324) 2024-04-24T01:58:09.311 INFO:tasks.rados.rados.0.smithi079.stdout:659: done (7 left) 2024-04-24T01:58:09.311 INFO:tasks.rados.rados.0.smithi079.stdout:660: done (6 left) 2024-04-24T01:58:09.311 INFO:tasks.rados.rados.0.smithi079.stdout:661: done (5 left) 2024-04-24T01:58:09.311 INFO:tasks.rados.rados.0.smithi079.stdout:670: done (4 left) 2024-04-24T01:58:09.311 INFO:tasks.rados.rados.0.smithi079.stdout:671: read oid 48 snap -1 2024-04-24T01:58:09.311 INFO:tasks.rados.rados.0.smithi079.stdout:671: expect (ObjNum 284 snap 75 seq_num 284) 2024-04-24T01:58:09.311 INFO:tasks.rados.rados.0.smithi079.stdout:672: delete oid 14 current snap is 79 2024-04-24T01:58:09.311 INFO:tasks.rados.rados.0.smithi079.stdout:665: expect (ObjNum 243 snap 67 seq_num 243) 2024-04-24T01:58:09.462 INFO:tasks.rados.rados.0.smithi079.stdout:665: done (5 left) 2024-04-24T01:58:09.462 INFO:tasks.rados.rados.0.smithi079.stdout:666: done (4 left) 2024-04-24T01:58:09.462 INFO:tasks.rados.rados.0.smithi079.stdout:667: done (3 left) 2024-04-24T01:58:09.462 INFO:tasks.rados.rados.0.smithi079.stdout:669: done (2 left) 2024-04-24T01:58:09.462 INFO:tasks.rados.rados.0.smithi079.stdout:672: done (1 left) 2024-04-24T01:58:09.463 INFO:tasks.rados.rados.0.smithi079.stdout:673: snap_remove snap 75 2024-04-24T01:58:09.690 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-04-24T01:58:09.690 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-04-24T01:58:09.690 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T01:58:09.798 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:58:09.793+0000 7f628e19d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:09.801 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:58:09.800+0000 7f628e19d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:09.987 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:58:09.988 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-24T01:58:10.000 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":252,"stamp":"2024-04-24T01:58:08.973652+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":112895435,"num_objects":106,"num_object_clones":60,"num_object_copies":318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":1359,"num_read_kb":864820,"num_write":1269,"num_write_kb":311849,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2298,"ondisk_log_size":2298,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":535588,"kb_used_data":411652,"kb_used_omap":0,"kb_used_meta":123904,"kb_avail":376951772,"statfs":{"total":386547056640,"available":385998614528,"internally_reserved":0,"allocated":421531648,"data_stored":478007569,"data_compressed":89889,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":126877696},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":7546409,"num_objects":-1,"num_object_clones":-1,"num_object_copies":-3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":-2,"num_read":19,"num_read_kb":17286,"num_write":20,"num_write_kb":5489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001076"},"pg_stats":[{"pgid":"3.f","version":"168'109","reported_seq":575,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-04-24T01:58:03.742523+0000","last_change":"2024-04-24T01:58:03.167502+0000","last_active":"2024-04-24T01:58:03.742523+0000","last_peered":"2024-04-24T01:58:03.742523+0000","last_clean":"2024-04-24T01:58:03.742523+0000","last_became_active":"2024-04-24T01:55:30.386726+0000","last_became_peered":"2024-04-24T01:55:30.386726+0000","last_unstale":"2024-04-24T01:58:03.742523+0000","last_undegraded":"2024-04-24T01:58:03.742523+0000","last_fullsized":"2024-04-24T01:58:03.742523+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":7,"snaptrim_duration":0,"stat_sum":{"num_bytes":6033779,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":68,"num_read_kb":56855,"num_write":53,"num_write_kb":15604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"168'179","reported_seq":640,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-04-24T01:58:03.705038+0000","last_change":"2024-04-24T01:58:03.771688+0000","last_active":"2024-04-24T01:58:03.705038+0000","last_peered":"2024-04-24T01:58:03.705038+0000","last_clean":"2024-04-24T01:58:03.705038+0000","last_became_active":"2024-04-24T01:55:30.321866+0000","last_became_peered":"2024-04-24T01:55:30.321866+0000","last_unstale":"2024-04-24T01:58:03.705038+0000","last_undegraded":"2024-04-24T01:58:03.705038+0000","last_fullsized":"2024-04-24T01:58:03.705038+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":179,"log_dups_size":0,"ondisk_log_size":179,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":9,"snaptrim_duration":0,"stat_sum":{"num_bytes":10689351,"num_objects":8,"num_object_clones":4,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":110,"num_read_kb":71606,"num_write":90,"num_write_kb":24583,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"173'166","reported_seq":629,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-04-24T01:58:08.125529+0000","last_change":"2024-04-24T01:58:08.149361+0000","last_active":"2024-04-24T01:58:08.125529+0000","last_peered":"2024-04-24T01:58:08.125529+0000","last_clean":"2024-04-24T01:58:08.125529+0000","last_became_active":"2024-04-24T01:55:30.320186+0000","last_became_peered":"2024-04-24T01:55:30.320186+0000","last_unstale":"2024-04-24T01:58:08.125529+0000","last_undegraded":"2024-04-24T01:58:08.125529+0000","last_fullsized":"2024-04-24T01:58:08.125529+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":166,"log_dups_size":0,"ondisk_log_size":166,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":13,"snaptrim_duration":0,"stat_sum":{"num_bytes":10033845,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":58,"num_read_kb":26139,"num_write":94,"num_write_kb":25777,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"168'149","reported_seq":614,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-04-24T01:58:03.131397+0000","last_change":"2024-04-24T01:58:03.170466+0000","last_active":"2024-04-24T01:58:03.131397+0000","last_peered":"2024-04-24T01:58:03.131397+0000","last_clean":"2024-04-24T01:58:03.131397+0000","last_became_active":"2024-04-24T01:55:30.288081+0000","last_became_peered":"2024-04-24T01:55:30.288081+0000","last_unstale":"2024-04-24T01:58:03.131397+0000","last_undegraded":"2024-04-24T01:58:03.131397+0000","last_fullsized":"2024-04-24T01:58:03.131397+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":10,"snaptrim_duration":0,"stat_sum":{"num_bytes":11916262,"num_objects":8,"num_object_clones":4,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":98,"num_read_kb":70485,"num_write":69,"num_write_kb":19743,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"162'58","reported_seq":534,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-04-24T01:58:04.107240+0000","last_change":"2024-04-24T01:58:04.119297+0000","last_active":"2024-04-24T01:58:04.107240+0000","last_peered":"2024-04-24T01:58:04.107240+0000","last_clean":"2024-04-24T01:58:04.107240+0000","last_became_active":"2024-04-24T01:55:30.333815+0000","last_became_peered":"2024-04-24T01:55:30.333815+0000","last_unstale":"2024-04-24T01:58:04.107240+0000","last_undegraded":"2024-04-24T01:58:04.107240+0000","last_fullsized":"2024-04-24T01:58:04.107240+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0,"stat_sum":{"num_bytes":3798382,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":32,"num_read_kb":17556,"num_write":36,"num_write_kb":10196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"168'289","reported_seq":744,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-04-24T01:58:03.975969+0000","last_change":"2024-04-24T01:58:03.227505+0000","last_active":"2024-04-24T01:58:03.975969+0000","last_peered":"2024-04-24T01:58:03.975969+0000","last_clean":"2024-04-24T01:58:03.975969+0000","last_became_active":"2024-04-24T01:55:30.315075+0000","last_became_peered":"2024-04-24T01:55:30.315075+0000","last_unstale":"2024-04-24T01:58:03.975969+0000","last_undegraded":"2024-04-24T01:58:03.975969+0000","last_fullsized":"2024-04-24T01:58:03.975969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":289,"log_dups_size":0,"ondisk_log_size":289,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":19,"snaptrim_duration":0,"stat_sum":{"num_bytes":11176688,"num_objects":17,"num_object_clones":11,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":165,"num_read_kb":102267,"num_write":149,"num_write_kb":39799,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"167'44","reported_seq":516,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-04-24T01:58:03.120868+0000","last_change":"2024-04-24T01:58:03.120868+0000","last_active":"2024-04-24T01:58:03.120868+0000","last_peered":"2024-04-24T01:58:03.120868+0000","last_clean":"2024-04-24T01:58:03.120868+0000","last_became_active":"2024-04-24T01:55:30.350300+0000","last_became_peered":"2024-04-24T01:55:30.350300+0000","last_unstale":"2024-04-24T01:58:03.120868+0000","last_undegraded":"2024-04-24T01:58:03.120868+0000","last_fullsized":"2024-04-24T01:58:03.120868+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":3,"snaptrim_duration":0,"stat_sum":{"num_bytes":3462242,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":26,"num_read_kb":13069,"num_write":36,"num_write_kb":8544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"163'8","reported_seq":489,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-04-24T01:58:08.131664+0000","last_change":"2024-04-24T01:58:08.143222+0000","last_active":"2024-04-24T01:58:08.131664+0000","last_peered":"2024-04-24T01:58:08.131664+0000","last_clean":"2024-04-24T01:58:08.131664+0000","last_became_active":"2024-04-24T01:55:30.376813+0000","last_became_peered":"2024-04-24T01:55:30.376813+0000","last_unstale":"2024-04-24T01:58:08.131664+0000","last_undegraded":"2024-04-24T01:58:08.131664+0000","last_fullsized":"2024-04-24T01:58:08.131664+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'97","reported_seq":547,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-04-24T01:58:04.097315+0000","last_change":"2024-04-24T01:55:16.196470+0000","last_active":"2024-04-24T01:58:04.097315+0000","last_peered":"2024-04-24T01:58:04.097315+0000","last_clean":"2024-04-24T01:58:04.097315+0000","last_became_active":"2024-04-24T01:55:16.190464+0000","last_became_peered":"2024-04-24T01:55:16.190464+0000","last_unstale":"2024-04-24T01:58:04.097315+0000","last_undegraded":"2024-04-24T01:58:04.097315+0000","last_fullsized":"2024-04-24T01:58:04.097315+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:15.127115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:15.127115+0000","last_clean_scrub_stamp":"2024-04-24T01:55:15.127115+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":418,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-04-24T01:58:08.125734+0000","last_change":"2024-04-24T01:55:18.236148+0000","last_active":"2024-04-24T01:58:08.125734+0000","last_peered":"2024-04-24T01:58:08.125734+0000","last_clean":"2024-04-24T01:58:08.125734+0000","last_became_active":"2024-04-24T01:55:18.230154+0000","last_became_peered":"2024-04-24T01:55:18.230154+0000","last_unstale":"2024-04-24T01:58:08.125734+0000","last_undegraded":"2024-04-24T01:58:08.125734+0000","last_fullsized":"2024-04-24T01:58:08.125734+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"173'241","reported_seq":725,"reported_epoch":173,"state":"active+clean+snaptrim","last_fresh":"2024-04-24T01:58:08.149901+0000","last_change":"2024-04-24T01:58:08.124783+0000","last_active":"2024-04-24T01:58:08.149901+0000","last_peered":"2024-04-24T01:58:08.149901+0000","last_clean":"2024-04-24T01:58:08.149901+0000","last_became_active":"2024-04-24T01:55:30.316222+0000","last_became_peered":"2024-04-24T01:55:30.316222+0000","last_unstale":"2024-04-24T01:58:08.149901+0000","last_undegraded":"2024-04-24T01:58:08.149901+0000","last_fullsized":"2024-04-24T01:58:08.149901+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":241,"log_dups_size":0,"ondisk_log_size":241,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":14,"snaptrim_duration":0,"stat_sum":{"num_bytes":11395212,"num_objects":10,"num_object_clones":6,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":146,"num_read_kb":98736,"num_write":126,"num_write_kb":34941,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":407,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-04-24T01:58:03.107158+0000","last_change":"2024-04-24T01:55:18.265905+0000","last_active":"2024-04-24T01:58:03.107158+0000","last_peered":"2024-04-24T01:58:03.107158+0000","last_clean":"2024-04-24T01:58:03.107158+0000","last_became_active":"2024-04-24T01:55:18.257020+0000","last_became_peered":"2024-04-24T01:55:18.257020+0000","last_unstale":"2024-04-24T01:58:03.107158+0000","last_undegraded":"2024-04-24T01:58:03.107158+0000","last_fullsized":"2024-04-24T01:58:03.107158+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"169'179","reported_seq":641,"reported_epoch":169,"state":"active+clean+snaptrim","last_fresh":"2024-04-24T01:58:04.120143+0000","last_change":"2024-04-24T01:58:04.100739+0000","last_active":"2024-04-24T01:58:04.120143+0000","last_peered":"2024-04-24T01:58:04.120143+0000","last_clean":"2024-04-24T01:58:04.120143+0000","last_became_active":"2024-04-24T01:55:30.344129+0000","last_became_peered":"2024-04-24T01:55:30.344129+0000","last_unstale":"2024-04-24T01:58:04.120143+0000","last_undegraded":"2024-04-24T01:58:04.120143+0000","last_fullsized":"2024-04-24T01:58:04.120143+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":179,"log_dups_size":0,"ondisk_log_size":179,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0,"stat_sum":{"num_bytes":6151428,"num_objects":11,"num_object_clones":8,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":88,"num_read_kb":53241,"num_write":91,"num_write_kb":26485,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":418,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-04-24T01:58:08.119129+0000","last_change":"2024-04-24T01:55:18.215251+0000","last_active":"2024-04-24T01:58:08.119129+0000","last_peered":"2024-04-24T01:58:08.119129+0000","last_clean":"2024-04-24T01:58:08.119129+0000","last_became_active":"2024-04-24T01:55:18.206605+0000","last_became_peered":"2024-04-24T01:55:18.206605+0000","last_unstale":"2024-04-24T01:58:08.119129+0000","last_undegraded":"2024-04-24T01:58:08.119129+0000","last_fullsized":"2024-04-24T01:58:08.119129+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"168'106","reported_seq":581,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-04-24T01:58:04.107396+0000","last_change":"2024-04-24T01:58:04.114803+0000","last_active":"2024-04-24T01:58:04.107396+0000","last_peered":"2024-04-24T01:58:04.107396+0000","last_clean":"2024-04-24T01:58:04.107396+0000","last_became_active":"2024-04-24T01:55:30.401366+0000","last_became_peered":"2024-04-24T01:55:30.401366+0000","last_unstale":"2024-04-24T01:58:04.107396+0000","last_undegraded":"2024-04-24T01:58:04.107396+0000","last_fullsized":"2024-04-24T01:58:04.107396+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":3,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655539,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":79,"num_read_kb":65582,"num_write":37,"num_write_kb":9692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":407,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-04-24T01:58:03.116194+0000","last_change":"2024-04-24T01:55:18.230893+0000","last_active":"2024-04-24T01:58:03.116194+0000","last_peered":"2024-04-24T01:58:03.116194+0000","last_clean":"2024-04-24T01:58:03.116194+0000","last_became_active":"2024-04-24T01:55:18.225188+0000","last_became_peered":"2024-04-24T01:55:18.225188+0000","last_unstale":"2024-04-24T01:58:03.116194+0000","last_undegraded":"2024-04-24T01:58:03.116194+0000","last_fullsized":"2024-04-24T01:58:03.116194+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"168'146","reported_seq":604,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-04-24T01:58:03.772017+0000","last_change":"2024-04-24T01:58:03.783184+0000","last_active":"2024-04-24T01:58:03.772017+0000","last_peered":"2024-04-24T01:58:03.772017+0000","last_clean":"2024-04-24T01:58:03.772017+0000","last_became_active":"2024-04-24T01:55:30.343318+0000","last_became_peered":"2024-04-24T01:55:30.343318+0000","last_unstale":"2024-04-24T01:58:03.772017+0000","last_undegraded":"2024-04-24T01:58:03.772017+0000","last_fullsized":"2024-04-24T01:58:03.772017+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":146,"log_dups_size":0,"ondisk_log_size":146,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":8,"snaptrim_duration":0,"stat_sum":{"num_bytes":8000132,"num_objects":9,"num_object_clones":6,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":84,"num_read_kb":72674,"num_write":78,"num_write_kb":22281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"173'230","reported_seq":695,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-04-24T01:58:08.149181+0000","last_change":"2024-04-24T01:58:08.172140+0000","last_active":"2024-04-24T01:58:08.149181+0000","last_peered":"2024-04-24T01:58:08.149181+0000","last_clean":"2024-04-24T01:58:08.149181+0000","last_became_active":"2024-04-24T01:55:30.345308+0000","last_became_peered":"2024-04-24T01:55:30.345308+0000","last_unstale":"2024-04-24T01:58:08.149181+0000","last_undegraded":"2024-04-24T01:58:08.149181+0000","last_fullsized":"2024-04-24T01:58:08.149181+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":230,"log_dups_size":0,"ondisk_log_size":230,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":12,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075813,"num_objects":9,"num_object_clones":5,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":124,"num_read_kb":84420,"num_write":96,"num_write_kb":23947,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":401,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-04-24T01:58:03.603616+0000","last_change":"2024-04-24T01:55:18.224238+0000","last_active":"2024-04-24T01:58:03.603616+0000","last_peered":"2024-04-24T01:58:03.603616+0000","last_clean":"2024-04-24T01:58:03.603616+0000","last_became_active":"2024-04-24T01:55:18.219131+0000","last_became_peered":"2024-04-24T01:55:18.219131+0000","last_unstale":"2024-04-24T01:58:03.603616+0000","last_undegraded":"2024-04-24T01:58:03.603616+0000","last_fullsized":"2024-04-24T01:58:03.603616+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"163'52","reported_seq":514,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-04-24T01:58:03.604676+0000","last_change":"2024-04-24T01:58:03.615123+0000","last_active":"2024-04-24T01:58:03.604676+0000","last_peered":"2024-04-24T01:58:03.604676+0000","last_clean":"2024-04-24T01:58:03.604676+0000","last_became_active":"2024-04-24T01:55:30.319958+0000","last_became_peered":"2024-04-24T01:55:30.319958+0000","last_unstale":"2024-04-24T01:58:03.604676+0000","last_undegraded":"2024-04-24T01:58:03.604676+0000","last_fullsized":"2024-04-24T01:58:03.604676+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0,"stat_sum":{"num_bytes":2823609,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":26,"num_read_kb":19608,"num_write":37,"num_write_kb":10363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":414,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-04-24T01:58:04.116138+0000","last_change":"2024-04-24T01:55:18.235763+0000","last_active":"2024-04-24T01:58:04.116138+0000","last_peered":"2024-04-24T01:58:04.116138+0000","last_clean":"2024-04-24T01:58:04.116138+0000","last_became_active":"2024-04-24T01:55:18.229175+0000","last_became_peered":"2024-04-24T01:55:18.229175+0000","last_unstale":"2024-04-24T01:58:04.116138+0000","last_undegraded":"2024-04-24T01:58:04.116138+0000","last_fullsized":"2024-04-24T01:58:04.116138+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"169'71","reported_seq":544,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-04-24T01:58:04.112722+0000","last_change":"2024-04-24T01:58:04.134110+0000","last_active":"2024-04-24T01:58:04.112722+0000","last_peered":"2024-04-24T01:58:04.112722+0000","last_clean":"2024-04-24T01:58:04.112722+0000","last_became_active":"2024-04-24T01:55:30.372690+0000","last_became_peered":"2024-04-24T01:55:30.372690+0000","last_unstale":"2024-04-24T01:58:04.112722+0000","last_undegraded":"2024-04-24T01:58:04.112722+0000","last_fullsized":"2024-04-24T01:58:04.112722+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":3,"snaptrim_duration":0,"stat_sum":{"num_bytes":6621991,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":39,"num_read_kb":24090,"num_write":38,"num_write_kb":11003,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":401,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-04-24T01:58:03.606780+0000","last_change":"2024-04-24T01:55:18.278351+0000","last_active":"2024-04-24T01:58:03.606780+0000","last_peered":"2024-04-24T01:58:03.606780+0000","last_clean":"2024-04-24T01:58:03.606780+0000","last_became_active":"2024-04-24T01:55:18.273668+0000","last_became_peered":"2024-04-24T01:55:18.273668+0000","last_unstale":"2024-04-24T01:58:03.606780+0000","last_undegraded":"2024-04-24T01:58:03.606780+0000","last_fullsized":"2024-04-24T01:58:03.606780+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"168'174","reported_seq":639,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-04-24T01:58:03.665562+0000","last_change":"2024-04-24T01:58:03.693752+0000","last_active":"2024-04-24T01:58:03.665562+0000","last_peered":"2024-04-24T01:58:03.665562+0000","last_clean":"2024-04-24T01:58:03.665562+0000","last_became_active":"2024-04-24T01:55:30.363123+0000","last_became_peered":"2024-04-24T01:55:30.363123+0000","last_unstale":"2024-04-24T01:58:03.665562+0000","last_undegraded":"2024-04-24T01:58:03.665562+0000","last_fullsized":"2024-04-24T01:58:03.665562+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":174,"log_dups_size":0,"ondisk_log_size":174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":14,"snaptrim_duration":0,"stat_sum":{"num_bytes":8470794,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":111,"num_read_kb":88400,"num_write":97,"num_write_kb":26149,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":418,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-04-24T01:58:08.127155+0000","last_change":"2024-04-24T01:55:18.241117+0000","last_active":"2024-04-24T01:58:08.127155+0000","last_peered":"2024-04-24T01:58:08.127155+0000","last_clean":"2024-04-24T01:58:08.127155+0000","last_became_active":"2024-04-24T01:55:18.234313+0000","last_became_peered":"2024-04-24T01:55:18.234313+0000","last_unstale":"2024-04-24T01:58:08.127155+0000","last_undegraded":"2024-04-24T01:58:08.127155+0000","last_fullsized":"2024-04-24T01:58:08.127155+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":112305067,"num_objects":104,"num_object_clones":60,"num_object_copies":312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":1256,"num_read_kb":864729,"num_write":1132,"num_write_kb":309955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":419672064,"data_stored":474386328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2201,"ondisk_log_size":2201,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":1}],"osd_stats":[{"osd":3,"up_from":10,"seq":42949673129,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":147580,"kb_used_data":116596,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94224260,"statfs":{"total":96636764160,"available":96485642240,"internally_reserved":0,"allocated":119394304,"data_stored":136930743,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673129,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":119260,"kb_used_data":88276,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94252580,"statfs":{"total":96636764160,"available":96514641920,"internally_reserved":0,"allocated":90394624,"data_stored":103253294,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705834,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":129540,"kb_used_data":98556,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94242300,"statfs":{"total":96636764160,"available":96504115200,"internally_reserved":0,"allocated":100921344,"data_stored":113815496,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705838,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":139208,"kb_used_data":108224,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94232632,"statfs":{"total":96636764160,"available":96494215168,"internally_reserved":0,"allocated":110821376,"data_stored":124008036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":102592512,"data_stored":116179820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":106946560,"data_stored":117789271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":89780224,"data_stored":102588257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":120352768,"data_stored":137828980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T01:58:10.115 INFO:tasks.rados.rados.0.smithi079.stdout:673: done (1 left) 2024-04-24T01:58:10.115 INFO:tasks.rados.rados.0.smithi079.stdout:674: snap_create 2024-04-24T01:58:10.115 INFO:tasks.rados.rados.0.smithi079.stdout:671: expect (ObjNum 284 snap 75 seq_num 284) 2024-04-24T01:58:10.909 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T01:58:11.015 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:58:11.013+0000 7efc5b8ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:11.019 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:58:11.018+0000 7efc5b8ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:11.115 INFO:tasks.rados.rados.0.smithi079.stdout:671: done (1 left) 2024-04-24T01:58:11.115 INFO:tasks.rados.rados.0.smithi079.stdout:674: done (0 left) 2024-04-24T01:58:11.115 INFO:tasks.rados.rados.0.smithi079.stdout:675: read oid 9 snap -1 2024-04-24T01:58:11.116 INFO:tasks.rados.rados.0.smithi079.stdout:675: expect (ObjNum 243 snap 67 seq_num 243) 2024-04-24T01:58:11.116 INFO:tasks.rados.rados.0.smithi079.stdout:676: delete oid 16 current snap is 80 2024-04-24T01:58:11.162 INFO:tasks.rados.rados.0.smithi079.stdout:675: expect (ObjNum 243 snap 67 seq_num 243) 2024-04-24T01:58:11.307 INFO:tasks.rados.rados.0.smithi079.stdout:675: done (1 left) 2024-04-24T01:58:11.307 INFO:tasks.rados.rados.0.smithi079.stdout:676: done (0 left) 2024-04-24T01:58:11.308 INFO:tasks.rados.rados.0.smithi079.stdout:677: read oid 21 snap 79 2024-04-24T01:58:11.308 INFO:tasks.rados.rados.0.smithi079.stdout:677: expect (ObjNum 266 snap 70 seq_num 266) 2024-04-24T01:58:11.308 INFO:tasks.rados.rados.0.smithi079.stdout:678: read oid 14 snap 53 2024-04-24T01:58:11.308 INFO:tasks.rados.rados.0.smithi079.stdout:678: expect (ObjNum 219 snap 49 seq_num 219) 2024-04-24T01:58:11.308 INFO:tasks.rados.rados.0.smithi079.stdout:679: snap_remove snap 71 2024-04-24T01:58:12.119 INFO:tasks.rados.rados.0.smithi079.stdout:679: done (2 left) 2024-04-24T01:58:12.119 INFO:tasks.rados.rados.0.smithi079.stdout:680: delete oid 4 current snap is 80 2024-04-24T01:58:12.119 INFO:tasks.rados.rados.0.smithi079.stdout:678: expect (ObjNum 219 snap 49 seq_num 219) 2024-04-24T01:58:12.455 INFO:tasks.rados.rados.0.smithi079.stdout:677: expect (ObjNum 266 snap 70 seq_num 266) 2024-04-24T01:58:12.582 INFO:tasks.rados.rados.0.smithi079.stdout:677: done (2 left) 2024-04-24T01:58:12.582 INFO:tasks.rados.rados.0.smithi079.stdout:678: done (1 left) 2024-04-24T01:58:12.582 INFO:tasks.rados.rados.0.smithi079.stdout:680: done (0 left) 2024-04-24T01:58:12.582 INFO:tasks.rados.rados.0.smithi079.stdout:681: snap_create 2024-04-24T01:58:13.112 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T01:58:13.121 INFO:tasks.rados.rados.0.smithi079.stdout:681: done (0 left) 2024-04-24T01:58:13.121 INFO:tasks.rados.rados.0.smithi079.stdout:682: write (excl) oid 1 current snap is 81 2024-04-24T01:58:13.121 INFO:tasks.rados.rados.0.smithi079.stdout:682: seq_num 332 ranges {779924=785847,2078257=580112,3301257=278667} 2024-04-24T01:58:13.130 INFO:tasks.rados.rados.0.smithi079.stdout:682: writing smithi07929762-1 from 779924 to 1565771 tid 1 2024-04-24T01:58:13.136 INFO:tasks.rados.rados.0.smithi079.stdout:682: writing smithi07929762-1 from 2078257 to 2658369 tid 2 2024-04-24T01:58:13.143 INFO:tasks.rados.rados.0.smithi079.stdout:682: writing smithi07929762-1 from 3301257 to 3579924 tid 3 2024-04-24T01:58:13.143 INFO:tasks.rados.rados.0.smithi079.stdout:683: write oid 23 current snap is 81 2024-04-24T01:58:13.143 INFO:tasks.rados.rados.0.smithi079.stdout:683: seq_num 333 ranges {796731=461096,1736739=723770,2928680=668051} 2024-04-24T01:58:13.152 INFO:tasks.rados.rados.0.smithi079.stdout:683: writing smithi07929762-23 from 796731 to 1257827 tid 1 2024-04-24T01:58:13.165 INFO:tasks.rados.rados.0.smithi079.stdout:683: writing smithi07929762-23 from 1736739 to 2460509 tid 2 2024-04-24T01:58:13.175 INFO:tasks.rados.rados.0.smithi079.stdout:683: writing smithi07929762-23 from 2928680 to 3596731 tid 3 2024-04-24T01:58:13.176 INFO:tasks.rados.rados.0.smithi079.stdout:684: write (excl) oid 18 current snap is 81 2024-04-24T01:58:13.176 INFO:tasks.rados.rados.0.smithi079.stdout:684: seq_num 334 ranges {413538=536345,1395221=583781,2556103=658617} 2024-04-24T01:58:13.184 INFO:tasks.rados.rados.0.smithi079.stdout:684: writing smithi07929762-18 from 413538 to 949883 tid 1 2024-04-24T01:58:13.192 INFO:tasks.rados.rados.0.smithi079.stdout:684: writing smithi07929762-18 from 1395221 to 1979002 tid 2 2024-04-24T01:58:13.203 INFO:tasks.rados.rados.0.smithi079.stdout:684: writing smithi07929762-18 from 2556103 to 3214720 tid 3 2024-04-24T01:58:13.203 INFO:tasks.rados.rados.0.smithi079.stdout:685: delete oid 41 current snap is 81 2024-04-24T01:58:13.209 INFO:tasks.rados.rados.0.smithi079.stdout:685: done (3 left) 2024-04-24T01:58:13.209 INFO:tasks.rados.rados.0.smithi079.stdout:686: write (excl) oid 45 current snap is 81 2024-04-24T01:58:13.209 INFO:tasks.rados.rados.0.smithi079.stdout:686: seq_num 336 ranges {447152=403196,1512185=587450,2727302=681161} 2024-04-24T01:58:13.216 INFO:tasks.rados.rados.0.smithi079.stdout:686: writing smithi07929762-45 from 447152 to 850348 tid 1 2024-04-24T01:58:13.225 INFO:tasks.rados.rados.0.smithi079.stdout:686: writing smithi07929762-45 from 1512185 to 2099635 tid 2 2024-04-24T01:58:13.236 INFO:tasks.rados.rados.0.smithi079.stdout:686: writing smithi07929762-45 from 2727302 to 3408463 tid 3 2024-04-24T01:58:13.236 INFO:tasks.rados.rados.0.smithi079.stdout:687: delete oid 26 current snap is 81 2024-04-24T01:58:13.237 INFO:tasks.rados.rados.0.smithi079.stdout:682: finishing write tid 1 to smithi07929762-1 2024-04-24T01:58:13.271 INFO:tasks.rados.rados.0.smithi079.stdout:683: finishing write tid 1 to smithi07929762-23 2024-04-24T01:58:13.272 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:58:13.269+0000 7fb66e39d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:13.277 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:58:13.276+0000 7fb66e39d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:13.285 INFO:tasks.rados.rados.0.smithi079.stdout:687: done (4 left) 2024-04-24T01:58:13.285 INFO:tasks.rados.rados.0.smithi079.stdout:688: snap_remove snap 70 2024-04-24T01:58:14.125 INFO:tasks.rados.rados.0.smithi079.stdout:688: done (4 left) 2024-04-24T01:58:14.125 INFO:tasks.rados.rados.0.smithi079.stdout:689: write oid 10 current snap is 81 2024-04-24T01:58:14.126 INFO:tasks.rados.rados.0.smithi079.stdout:683: finishing write tid 2 to smithi07929762-23 2024-04-24T01:58:14.126 INFO:tasks.rados.rados.0.smithi079.stdout:689: seq_num 338 ranges {480766=553694,1629149=591119,2782148=420058} 2024-04-24T01:58:14.131 INFO:tasks.rados.rados.0.smithi079.stdout:689: writing smithi07929762-10 from 480766 to 1034460 tid 1 2024-04-24T01:58:14.137 INFO:tasks.rados.rados.0.smithi079.stdout:689: writing smithi07929762-10 from 1629149 to 2220268 tid 2 2024-04-24T01:58:14.144 INFO:tasks.rados.rados.0.smithi079.stdout:689: writing smithi07929762-10 from 2782148 to 3202206 tid 3 2024-04-24T01:58:14.144 INFO:tasks.rados.rados.0.smithi079.stdout:690: snap_remove snap 80 2024-04-24T01:58:15.002 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-04-24T01:58:15.002 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-04-24T01:58:15.003 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T01:58:15.111 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:58:15.106+0000 7f46cd220640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:15.115 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:58:15.114+0000 7f46cd220640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:15.131 INFO:tasks.rados.rados.0.smithi079.stdout:690: done (5 left) 2024-04-24T01:58:15.131 INFO:tasks.rados.rados.0.smithi079.stdout:691: snap_create 2024-04-24T01:58:15.131 INFO:tasks.rados.rados.0.smithi079.stdout:686: finishing write tid 1 to smithi07929762-45 2024-04-24T01:58:15.131 INFO:tasks.rados.rados.0.smithi079.stdout:683: finishing write tid 3 to smithi07929762-23 2024-04-24T01:58:15.131 INFO:tasks.rados.rados.0.smithi079.stdout:683: finishing write tid 4 to smithi07929762-23 2024-04-24T01:58:15.132 INFO:tasks.rados.rados.0.smithi079.stdout:683: finishing write tid 5 to smithi07929762-23 2024-04-24T01:58:15.132 INFO:tasks.rados.rados.0.smithi079.stdout:683: oid 23 updating version 0 to 58 2024-04-24T01:58:15.132 INFO:tasks.rados.rados.0.smithi079.stdout:683: oid 23 version 58 is already newer than 58 2024-04-24T01:58:15.132 INFO:tasks.rados.rados.0.smithi079.stdout:683: oid 23 version 58 is already newer than 58 2024-04-24T01:58:15.132 INFO:tasks.rados.rados.0.smithi079.stdout:683: oid 23 version 58 is already newer than 58 2024-04-24T01:58:15.132 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 23 v 58 (ObjNum 333 snap 81 seq_num 333) dirty exists 2024-04-24T01:58:15.132 INFO:tasks.rados.rados.0.smithi079.stdout:683: left oid 23 (ObjNum 333 snap 81 seq_num 333) 2024-04-24T01:58:15.132 INFO:tasks.rados.rados.0.smithi079.stdout:682: finishing write tid 2 to smithi07929762-1 2024-04-24T01:58:15.132 INFO:tasks.rados.rados.0.smithi079.stdout:686: finishing write tid 2 to smithi07929762-45 2024-04-24T01:58:15.132 INFO:tasks.rados.rados.0.smithi079.stdout:682: finishing write tid 3 to smithi07929762-1 2024-04-24T01:58:15.132 INFO:tasks.rados.rados.0.smithi079.stdout:682: finishing write tid 4 to smithi07929762-1 2024-04-24T01:58:15.132 INFO:tasks.rados.rados.0.smithi079.stdout:686: finishing write tid 3 to smithi07929762-45 2024-04-24T01:58:15.132 INFO:tasks.rados.rados.0.smithi079.stdout:682: finishing write tid 5 to smithi07929762-1 2024-04-24T01:58:15.132 INFO:tasks.rados.rados.0.smithi079.stdout:682: oid 1 updating version 0 to 368 2024-04-24T01:58:15.132 INFO:tasks.rados.rados.0.smithi079.stdout:682: oid 1 version 368 is already newer than 365 2024-04-24T01:58:15.133 INFO:tasks.rados.rados.0.smithi079.stdout:682: oid 1 version 368 is already newer than 368 2024-04-24T01:58:15.133 INFO:tasks.rados.rados.0.smithi079.stdout:682: oid 1 version 368 is already newer than 368 2024-04-24T01:58:15.133 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 1 v 368 (ObjNum 332 snap 81 seq_num 332) dirty exists 2024-04-24T01:58:15.133 INFO:tasks.rados.rados.0.smithi079.stdout:682: left oid 1 (ObjNum 332 snap 81 seq_num 332) 2024-04-24T01:58:15.133 INFO:tasks.rados.rados.0.smithi079.stdout:686: finishing write tid 4 to smithi07929762-45 2024-04-24T01:58:15.133 INFO:tasks.rados.rados.0.smithi079.stdout:686: finishing write tid 5 to smithi07929762-45 2024-04-24T01:58:15.133 INFO:tasks.rados.rados.0.smithi079.stdout:686: oid 45 updating version 0 to 200 2024-04-24T01:58:15.133 INFO:tasks.rados.rados.0.smithi079.stdout:686: oid 45 version 200 is already newer than 200 2024-04-24T01:58:15.133 INFO:tasks.rados.rados.0.smithi079.stdout:686: oid 45 version 200 is already newer than 197 2024-04-24T01:58:15.133 INFO:tasks.rados.rados.0.smithi079.stdout:686: oid 45 version 200 is already newer than 200 2024-04-24T01:58:15.133 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 45 v 200 (ObjNum 336 snap 81 seq_num 336) dirty exists 2024-04-24T01:58:15.133 INFO:tasks.rados.rados.0.smithi079.stdout:686: left oid 45 (ObjNum 336 snap 81 seq_num 336) 2024-04-24T01:58:15.133 INFO:tasks.rados.rados.0.smithi079.stdout:684: finishing write tid 1 to smithi07929762-18 2024-04-24T01:58:15.133 INFO:tasks.rados.rados.0.smithi079.stdout:684: finishing write tid 2 to smithi07929762-18 2024-04-24T01:58:15.133 INFO:tasks.rados.rados.0.smithi079.stdout:684: finishing write tid 3 to smithi07929762-18 2024-04-24T01:58:15.133 INFO:tasks.rados.rados.0.smithi079.stdout:684: finishing write tid 4 to smithi07929762-18 2024-04-24T01:58:15.133 INFO:tasks.rados.rados.0.smithi079.stdout:684: finishing write tid 5 to smithi07929762-18 2024-04-24T01:58:15.134 INFO:tasks.rados.rados.0.smithi079.stdout:684: oid 18 updating version 0 to 372 2024-04-24T01:58:15.134 INFO:tasks.rados.rados.0.smithi079.stdout:684: oid 18 version 372 is already newer than 372 2024-04-24T01:58:15.134 INFO:tasks.rados.rados.0.smithi079.stdout:684: oid 18 version 372 is already newer than 369 2024-04-24T01:58:15.134 INFO:tasks.rados.rados.0.smithi079.stdout:684: oid 18 version 372 is already newer than 372 2024-04-24T01:58:15.134 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 18 v 372 (ObjNum 334 snap 81 seq_num 334) dirty exists 2024-04-24T01:58:15.134 INFO:tasks.rados.rados.0.smithi079.stdout:684: left oid 18 (ObjNum 334 snap 81 seq_num 334) 2024-04-24T01:58:15.134 INFO:tasks.rados.rados.0.smithi079.stdout:689: finishing write tid 1 to smithi07929762-10 2024-04-24T01:58:15.134 INFO:tasks.rados.rados.0.smithi079.stdout:689: finishing write tid 2 to smithi07929762-10 2024-04-24T01:58:15.134 INFO:tasks.rados.rados.0.smithi079.stdout:689: finishing write tid 3 to smithi07929762-10 2024-04-24T01:58:15.134 INFO:tasks.rados.rados.0.smithi079.stdout:689: finishing write tid 4 to smithi07929762-10 2024-04-24T01:58:15.134 INFO:tasks.rados.rados.0.smithi079.stdout:689: finishing write tid 5 to smithi07929762-10 2024-04-24T01:58:15.134 INFO:tasks.rados.rados.0.smithi079.stdout:689: oid 10 updating version 0 to 207 2024-04-24T01:58:15.134 INFO:tasks.rados.rados.0.smithi079.stdout:689: oid 10 version 207 is already newer than 207 2024-04-24T01:58:15.134 INFO:tasks.rados.rados.0.smithi079.stdout:689: oid 10 version 207 is already newer than 207 2024-04-24T01:58:15.135 INFO:tasks.rados.rados.0.smithi079.stdout:689: oid 10 version 207 is already newer than 207 2024-04-24T01:58:15.135 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 10 v 207 (ObjNum 338 snap 81 seq_num 338) dirty exists 2024-04-24T01:58:15.135 INFO:tasks.rados.rados.0.smithi079.stdout:689: left oid 10 (ObjNum 338 snap 81 seq_num 338) 2024-04-24T01:58:15.313 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:58:15.313 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-24T01:58:15.325 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":261,"stamp":"2024-04-24T01:58:15.128266+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":106525417,"num_objects":100,"num_object_clones":55,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":1423,"num_read_kb":922973,"num_write":1300,"num_write_kb":320058,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2575,"ondisk_log_size":2575,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":509816,"kb_used_data":385600,"kb_used_omap":0,"kb_used_meta":123904,"kb_avail":376977544,"statfs":{"total":386547056640,"available":386025005056,"internally_reserved":0,"allocated":394854400,"data_stored":437168132,"data_compressed":89889,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":126877696},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-1495790,"num_objects":-2,"num_object_clones":-1,"num_object_copies":-6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":38,"num_read_kb":30362,"num_write":23,"num_write_kb":6792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.014189"},"pg_stats":[{"pgid":"3.f","version":"176'121","reported_seq":621,"reported_epoch":178,"state":"active+clean+snaptrim","last_fresh":"2024-04-24T01:58:14.149643+0000","last_change":"2024-04-24T01:58:14.149643+0000","last_active":"2024-04-24T01:58:14.149643+0000","last_peered":"2024-04-24T01:58:14.149643+0000","last_clean":"2024-04-24T01:58:14.149643+0000","last_became_active":"2024-04-24T01:55:30.386726+0000","last_became_peered":"2024-04-24T01:55:30.386726+0000","last_unstale":"2024-04-24T01:58:14.149643+0000","last_undegraded":"2024-04-24T01:58:14.149643+0000","last_fullsized":"2024-04-24T01:58:14.149643+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":8,"snaptrim_duration":0,"stat_sum":{"num_bytes":6689219,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":74,"num_read_kb":63446,"num_write":53,"num_write_kb":15604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"177'200","reported_seq":691,"reported_epoch":178,"state":"active+clean+snaptrim","last_fresh":"2024-04-24T01:58:14.148339+0000","last_change":"2024-04-24T01:58:14.148339+0000","last_active":"2024-04-24T01:58:14.148339+0000","last_peered":"2024-04-24T01:58:14.148339+0000","last_clean":"2024-04-24T01:58:14.148339+0000","last_became_active":"2024-04-24T01:55:30.321866+0000","last_became_peered":"2024-04-24T01:55:30.321866+0000","last_unstale":"2024-04-24T01:58:14.148339+0000","last_undegraded":"2024-04-24T01:58:14.148339+0000","last_fullsized":"2024-04-24T01:58:14.148339+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":10,"snaptrim_duration":0,"stat_sum":{"num_bytes":11613736,"num_objects":8,"num_object_clones":4,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":118,"num_read_kb":75435,"num_write":95,"num_write_kb":26217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"176'176","reported_seq":651,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-04-24T01:58:13.161768+0000","last_change":"2024-04-24T01:58:12.272508+0000","last_active":"2024-04-24T01:58:13.161768+0000","last_peered":"2024-04-24T01:58:13.161768+0000","last_clean":"2024-04-24T01:58:13.161768+0000","last_became_active":"2024-04-24T01:55:30.320186+0000","last_became_peered":"2024-04-24T01:55:30.320186+0000","last_unstale":"2024-04-24T01:58:13.161768+0000","last_undegraded":"2024-04-24T01:58:13.161768+0000","last_fullsized":"2024-04-24T01:58:13.161768+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":176,"log_dups_size":0,"ondisk_log_size":176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":13,"snaptrim_duration":0,"stat_sum":{"num_bytes":6353112,"num_objects":6,"num_object_clones":3,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":64,"num_read_kb":35668,"num_write":95,"num_write_kb":25777,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"176'172","reported_seq":668,"reported_epoch":178,"state":"active+clean+snaptrim","last_fresh":"2024-04-24T01:58:14.152321+0000","last_change":"2024-04-24T01:58:14.152321+0000","last_active":"2024-04-24T01:58:14.152321+0000","last_peered":"2024-04-24T01:58:14.152321+0000","last_clean":"2024-04-24T01:58:14.152321+0000","last_became_active":"2024-04-24T01:55:30.288081+0000","last_became_peered":"2024-04-24T01:55:30.288081+0000","last_unstale":"2024-04-24T01:58:14.152321+0000","last_undegraded":"2024-04-24T01:58:14.152321+0000","last_fullsized":"2024-04-24T01:58:14.152321+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":172,"log_dups_size":0,"ondisk_log_size":172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":11,"snaptrim_duration":0,"stat_sum":{"num_bytes":13159980,"num_objects":8,"num_object_clones":4,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":105,"num_read_kb":77553,"num_write":74,"num_write_kb":21160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"173'59","reported_seq":562,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-04-24T01:58:13.167400+0000","last_change":"2024-04-24T01:58:12.198208+0000","last_active":"2024-04-24T01:58:13.167400+0000","last_peered":"2024-04-24T01:58:13.167400+0000","last_clean":"2024-04-24T01:58:13.167400+0000","last_became_active":"2024-04-24T01:55:30.333815+0000","last_became_peered":"2024-04-24T01:55:30.333815+0000","last_unstale":"2024-04-24T01:58:13.167400+0000","last_undegraded":"2024-04-24T01:58:13.167400+0000","last_fullsized":"2024-04-24T01:58:13.167400+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0,"stat_sum":{"num_bytes":3798382,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":32,"num_read_kb":17556,"num_write":36,"num_write_kb":10196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"177'372","reported_seq":829,"reported_epoch":178,"state":"active+clean+snaptrim","last_fresh":"2024-04-24T01:58:14.156709+0000","last_change":"2024-04-24T01:58:14.156709+0000","last_active":"2024-04-24T01:58:14.156709+0000","last_peered":"2024-04-24T01:58:14.156709+0000","last_clean":"2024-04-24T01:58:14.156709+0000","last_became_active":"2024-04-24T01:55:30.315075+0000","last_became_peered":"2024-04-24T01:55:30.315075+0000","last_unstale":"2024-04-24T01:58:14.156709+0000","last_undegraded":"2024-04-24T01:58:14.156709+0000","last_fullsized":"2024-04-24T01:58:14.156709+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":372,"log_dups_size":0,"ondisk_log_size":372,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":21,"snaptrim_duration":0,"stat_sum":{"num_bytes":9512762,"num_objects":17,"num_object_clones":11,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":169,"num_read_kb":102269,"num_write":162,"num_write_kb":43146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"177'58","reported_seq":562,"reported_epoch":178,"state":"active+clean+snaptrim","last_fresh":"2024-04-24T01:58:14.159153+0000","last_change":"2024-04-24T01:58:14.159153+0000","last_active":"2024-04-24T01:58:14.159153+0000","last_peered":"2024-04-24T01:58:14.159153+0000","last_clean":"2024-04-24T01:58:14.159153+0000","last_became_active":"2024-04-24T01:55:30.350300+0000","last_became_peered":"2024-04-24T01:55:30.350300+0000","last_unstale":"2024-04-24T01:58:14.159153+0000","last_undegraded":"2024-04-24T01:58:14.159153+0000","last_fullsized":"2024-04-24T01:58:14.159153+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":5,"snaptrim_duration":0,"stat_sum":{"num_bytes":3596731,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":27,"num_read_kb":13070,"num_write":41,"num_write_kb":10355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"163'8","reported_seq":502,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-04-24T01:58:13.189431+0000","last_change":"2024-04-24T01:58:12.186669+0000","last_active":"2024-04-24T01:58:13.189431+0000","last_peered":"2024-04-24T01:58:13.189431+0000","last_clean":"2024-04-24T01:58:13.189431+0000","last_became_active":"2024-04-24T01:55:30.376813+0000","last_became_peered":"2024-04-24T01:55:30.376813+0000","last_unstale":"2024-04-24T01:58:13.189431+0000","last_undegraded":"2024-04-24T01:58:13.189431+0000","last_fullsized":"2024-04-24T01:58:13.189431+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'97","reported_seq":569,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-04-24T01:58:13.158463+0000","last_change":"2024-04-24T01:55:16.196470+0000","last_active":"2024-04-24T01:58:13.158463+0000","last_peered":"2024-04-24T01:58:13.158463+0000","last_clean":"2024-04-24T01:58:13.158463+0000","last_became_active":"2024-04-24T01:55:16.190464+0000","last_became_peered":"2024-04-24T01:55:16.190464+0000","last_unstale":"2024-04-24T01:58:13.158463+0000","last_undegraded":"2024-04-24T01:58:13.158463+0000","last_fullsized":"2024-04-24T01:58:13.158463+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:15.127115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:15.127115+0000","last_clean_scrub_stamp":"2024-04-24T01:55:15.127115+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":430,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-04-24T01:58:13.176747+0000","last_change":"2024-04-24T01:55:18.236148+0000","last_active":"2024-04-24T01:58:13.176747+0000","last_peered":"2024-04-24T01:58:13.176747+0000","last_clean":"2024-04-24T01:58:13.176747+0000","last_became_active":"2024-04-24T01:55:18.230154+0000","last_became_peered":"2024-04-24T01:55:18.230154+0000","last_unstale":"2024-04-24T01:58:13.176747+0000","last_undegraded":"2024-04-24T01:58:13.176747+0000","last_fullsized":"2024-04-24T01:58:13.176747+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"176'257","reported_seq":745,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-04-24T01:58:12.303370+0000","last_change":"2024-04-24T01:58:12.303370+0000","last_active":"2024-04-24T01:58:12.303370+0000","last_peered":"2024-04-24T01:58:12.303370+0000","last_clean":"2024-04-24T01:58:12.303370+0000","last_became_active":"2024-04-24T01:55:30.316222+0000","last_became_peered":"2024-04-24T01:55:30.316222+0000","last_unstale":"2024-04-24T01:58:12.303370+0000","last_undegraded":"2024-04-24T01:58:12.303370+0000","last_fullsized":"2024-04-24T01:58:12.303370+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":257,"log_dups_size":0,"ondisk_log_size":257,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":14,"snaptrim_duration":0,"stat_sum":{"num_bytes":11395212,"num_objects":10,"num_object_clones":6,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":146,"num_read_kb":98736,"num_write":126,"num_write_kb":34941,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":437,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-24T01:58:14.164979+0000","last_change":"2024-04-24T01:55:18.265905+0000","last_active":"2024-04-24T01:58:14.164979+0000","last_peered":"2024-04-24T01:58:14.164979+0000","last_clean":"2024-04-24T01:58:14.164979+0000","last_became_active":"2024-04-24T01:55:18.257020+0000","last_became_peered":"2024-04-24T01:55:18.257020+0000","last_unstale":"2024-04-24T01:58:14.164979+0000","last_undegraded":"2024-04-24T01:58:14.164979+0000","last_fullsized":"2024-04-24T01:58:14.164979+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"176'213","reported_seq":689,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-04-24T01:58:13.163175+0000","last_change":"2024-04-24T01:58:12.301640+0000","last_active":"2024-04-24T01:58:13.163175+0000","last_peered":"2024-04-24T01:58:13.163175+0000","last_clean":"2024-04-24T01:58:13.163175+0000","last_became_active":"2024-04-24T01:55:30.344129+0000","last_became_peered":"2024-04-24T01:55:30.344129+0000","last_unstale":"2024-04-24T01:58:13.163175+0000","last_undegraded":"2024-04-24T01:58:13.163175+0000","last_fullsized":"2024-04-24T01:58:13.163175+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":213,"log_dups_size":0,"ondisk_log_size":213,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":9,"snaptrim_duration":0,"stat_sum":{"num_bytes":6151428,"num_objects":8,"num_object_clones":5,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":94,"num_read_kb":60231,"num_write":91,"num_write_kb":26485,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":430,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-04-24T01:58:13.168134+0000","last_change":"2024-04-24T01:55:18.215251+0000","last_active":"2024-04-24T01:58:13.168134+0000","last_peered":"2024-04-24T01:58:13.168134+0000","last_clean":"2024-04-24T01:58:13.168134+0000","last_became_active":"2024-04-24T01:55:18.206605+0000","last_became_peered":"2024-04-24T01:55:18.206605+0000","last_unstale":"2024-04-24T01:58:13.168134+0000","last_undegraded":"2024-04-24T01:58:13.168134+0000","last_fullsized":"2024-04-24T01:58:13.168134+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"176'112","reported_seq":613,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-04-24T01:58:13.172033+0000","last_change":"2024-04-24T01:58:12.270641+0000","last_active":"2024-04-24T01:58:13.172033+0000","last_peered":"2024-04-24T01:58:13.172033+0000","last_clean":"2024-04-24T01:58:13.172033+0000","last_became_active":"2024-04-24T01:55:30.401366+0000","last_became_peered":"2024-04-24T01:55:30.401366+0000","last_unstale":"2024-04-24T01:58:13.172033+0000","last_undegraded":"2024-04-24T01:58:13.172033+0000","last_fullsized":"2024-04-24T01:58:13.172033+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0,"stat_sum":{"num_bytes":4504243,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":80,"num_read_kb":65582,"num_write":37,"num_write_kb":9692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":437,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-24T01:58:14.166724+0000","last_change":"2024-04-24T01:55:18.230893+0000","last_active":"2024-04-24T01:58:14.166724+0000","last_peered":"2024-04-24T01:58:14.166724+0000","last_clean":"2024-04-24T01:58:14.166724+0000","last_became_active":"2024-04-24T01:55:18.225188+0000","last_became_peered":"2024-04-24T01:55:18.225188+0000","last_unstale":"2024-04-24T01:58:14.166724+0000","last_undegraded":"2024-04-24T01:58:14.166724+0000","last_fullsized":"2024-04-24T01:58:14.166724+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"176'180","reported_seq":659,"reported_epoch":178,"state":"active+clean+snaptrim","last_fresh":"2024-04-24T01:58:14.150255+0000","last_change":"2024-04-24T01:58:14.150255+0000","last_active":"2024-04-24T01:58:14.150255+0000","last_peered":"2024-04-24T01:58:14.150255+0000","last_clean":"2024-04-24T01:58:14.150255+0000","last_became_active":"2024-04-24T01:55:30.343318+0000","last_became_peered":"2024-04-24T01:55:30.343318+0000","last_unstale":"2024-04-24T01:58:14.150255+0000","last_undegraded":"2024-04-24T01:58:14.150255+0000","last_fullsized":"2024-04-24T01:58:14.150255+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":180,"log_dups_size":0,"ondisk_log_size":180,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":9,"snaptrim_duration":0,"stat_sum":{"num_bytes":8000132,"num_objects":8,"num_object_clones":5,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":90,"num_read_kb":79021,"num_write":78,"num_write_kb":22281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"176'235","reported_seq":716,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-04-24T01:58:12.252471+0000","last_change":"2024-04-24T01:58:12.252471+0000","last_active":"2024-04-24T01:58:12.252471+0000","last_peered":"2024-04-24T01:58:12.252471+0000","last_clean":"2024-04-24T01:58:12.252471+0000","last_became_active":"2024-04-24T01:55:30.345308+0000","last_became_peered":"2024-04-24T01:55:30.345308+0000","last_unstale":"2024-04-24T01:58:12.252471+0000","last_undegraded":"2024-04-24T01:58:12.252471+0000","last_fullsized":"2024-04-24T01:58:12.252471+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":235,"log_dups_size":0,"ondisk_log_size":235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":12,"snaptrim_duration":0,"stat_sum":{"num_bytes":5764801,"num_objects":9,"num_object_clones":5,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":136,"num_read_kb":94830,"num_write":97,"num_write_kb":23947,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":429,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-24T01:58:14.159343+0000","last_change":"2024-04-24T01:55:18.224238+0000","last_active":"2024-04-24T01:58:14.159343+0000","last_peered":"2024-04-24T01:58:14.159343+0000","last_clean":"2024-04-24T01:58:14.159343+0000","last_became_active":"2024-04-24T01:55:18.219131+0000","last_became_peered":"2024-04-24T01:55:18.219131+0000","last_unstale":"2024-04-24T01:58:14.159343+0000","last_undegraded":"2024-04-24T01:58:14.159343+0000","last_fullsized":"2024-04-24T01:58:14.159343+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"174'54","reported_seq":550,"reported_epoch":178,"state":"active+clean+snaptrim","last_fresh":"2024-04-24T01:58:14.159449+0000","last_change":"2024-04-24T01:58:14.159449+0000","last_active":"2024-04-24T01:58:14.159449+0000","last_peered":"2024-04-24T01:58:14.159449+0000","last_clean":"2024-04-24T01:58:14.159449+0000","last_became_active":"2024-04-24T01:55:30.319958+0000","last_became_peered":"2024-04-24T01:55:30.319958+0000","last_unstale":"2024-04-24T01:58:14.159449+0000","last_undegraded":"2024-04-24T01:58:14.159449+0000","last_fullsized":"2024-04-24T01:58:14.159449+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":5,"snaptrim_duration":0,"stat_sum":{"num_bytes":2823609,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":26,"num_read_kb":19608,"num_write":37,"num_write_kb":10363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":436,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-04-24T01:58:13.171050+0000","last_change":"2024-04-24T01:55:18.235763+0000","last_active":"2024-04-24T01:58:13.171050+0000","last_peered":"2024-04-24T01:58:13.171050+0000","last_clean":"2024-04-24T01:58:13.171050+0000","last_became_active":"2024-04-24T01:55:18.229175+0000","last_became_peered":"2024-04-24T01:55:18.229175+0000","last_unstale":"2024-04-24T01:58:13.171050+0000","last_undegraded":"2024-04-24T01:58:13.171050+0000","last_fullsized":"2024-04-24T01:58:13.171050+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"176'77","reported_seq":575,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-04-24T01:58:13.164333+0000","last_change":"2024-04-24T01:58:12.271047+0000","last_active":"2024-04-24T01:58:13.164333+0000","last_peered":"2024-04-24T01:58:13.164333+0000","last_clean":"2024-04-24T01:58:13.164333+0000","last_became_active":"2024-04-24T01:55:30.372690+0000","last_became_peered":"2024-04-24T01:55:30.372690+0000","last_unstale":"2024-04-24T01:58:13.164333+0000","last_undegraded":"2024-04-24T01:58:13.164333+0000","last_fullsized":"2024-04-24T01:58:13.164333+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":3,"snaptrim_duration":0,"stat_sum":{"num_bytes":6621991,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":39,"num_read_kb":24090,"num_write":38,"num_write_kb":11003,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":429,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-24T01:58:14.157772+0000","last_change":"2024-04-24T01:55:18.278351+0000","last_active":"2024-04-24T01:58:14.157772+0000","last_peered":"2024-04-24T01:58:14.157772+0000","last_clean":"2024-04-24T01:58:14.157772+0000","last_became_active":"2024-04-24T01:55:18.273668+0000","last_became_peered":"2024-04-24T01:55:18.273668+0000","last_unstale":"2024-04-24T01:58:14.157772+0000","last_undegraded":"2024-04-24T01:58:14.157772+0000","last_fullsized":"2024-04-24T01:58:14.157772+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"176'184","reported_seq":682,"reported_epoch":178,"state":"active+clean+snaptrim","last_fresh":"2024-04-24T01:58:14.165519+0000","last_change":"2024-04-24T01:58:14.165519+0000","last_active":"2024-04-24T01:58:14.165519+0000","last_peered":"2024-04-24T01:58:14.165519+0000","last_clean":"2024-04-24T01:58:14.165519+0000","last_became_active":"2024-04-24T01:55:30.363123+0000","last_became_peered":"2024-04-24T01:55:30.363123+0000","last_unstale":"2024-04-24T01:58:14.165519+0000","last_undegraded":"2024-04-24T01:58:14.165519+0000","last_fullsized":"2024-04-24T01:58:14.165519+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":184,"log_dups_size":0,"ondisk_log_size":184,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":14,"snaptrim_duration":0,"stat_sum":{"num_bytes":5949711,"num_objects":6,"num_object_clones":3,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":118,"num_read_kb":95786,"num_write":98,"num_write_kb":26149,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":428,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-04-24T01:58:12.176250+0000","last_change":"2024-04-24T01:55:18.241117+0000","last_active":"2024-04-24T01:58:12.176250+0000","last_peered":"2024-04-24T01:58:12.176250+0000","last_clean":"2024-04-24T01:58:12.176250+0000","last_became_active":"2024-04-24T01:55:18.234313+0000","last_became_peered":"2024-04-24T01:55:18.234313+0000","last_unstale":"2024-04-24T01:58:12.176250+0000","last_undegraded":"2024-04-24T01:58:12.176250+0000","last_fullsized":"2024-04-24T01:58:12.176250+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":105935049,"num_objects":98,"num_object_clones":55,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":1320,"num_read_kb":922882,"num_write":1163,"num_write_kb":318164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393875456,"data_stored":436191553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2478,"ondisk_log_size":2478,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":1}],"osd_stats":[{"osd":3,"up_from":10,"seq":42949673139,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":141840,"kb_used_data":110856,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94230000,"statfs":{"total":96636764160,"available":96491520000,"internally_reserved":0,"allocated":113516544,"data_stored":126883158,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673139,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":113280,"kb_used_data":82016,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94258560,"statfs":{"total":96636764160,"available":96520765440,"internally_reserved":0,"allocated":83984384,"data_stored":94148598,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705844,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":125904,"kb_used_data":94920,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94245936,"statfs":{"total":96636764160,"available":96507838464,"internally_reserved":0,"allocated":97198080,"data_stored":107809177,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705843,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":128792,"kb_used_data":97808,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94243048,"statfs":{"total":96636764160,"available":96504881152,"internally_reserved":0,"allocated":100155392,"data_stored":108327199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":97472512,"data_stored":108370167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":99500032,"data_stored":108489298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":83623936,"data_stored":92703914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":113278976,"data_stored":126628174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T01:58:15.419 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T01:58:15.526 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:58:15.521+0000 7f9a686ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:15.527 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:58:15.526+0000 7f9a686ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:16.037 INFO:tasks.rados.rados.0.smithi079.stdout:682: done (5 left) 2024-04-24T01:58:16.037 INFO:tasks.rados.rados.0.smithi079.stdout:683: done (4 left) 2024-04-24T01:58:16.037 INFO:tasks.rados.rados.0.smithi079.stdout:684: done (3 left) 2024-04-24T01:58:16.037 INFO:tasks.rados.rados.0.smithi079.stdout:686: done (2 left) 2024-04-24T01:58:16.037 INFO:tasks.rados.rados.0.smithi079.stdout:689: done (1 left) 2024-04-24T01:58:16.037 INFO:tasks.rados.rados.0.smithi079.stdout:691: done (0 left) 2024-04-24T01:58:16.037 INFO:tasks.rados.rados.0.smithi079.stdout:692: read oid 19 snap -1 2024-04-24T01:58:16.037 INFO:tasks.rados.rados.0.smithi079.stdout:692: expect deleted 2024-04-24T01:58:16.037 INFO:tasks.rados.rados.0.smithi079.stdout:693: read oid 30 snap -1 2024-04-24T01:58:16.037 INFO:tasks.rados.rados.0.smithi079.stdout:693: expect (ObjNum 290 snap 76 seq_num 290) 2024-04-24T01:58:16.037 INFO:tasks.rados.rados.0.smithi079.stdout:694: delete oid 42 current snap is 82 2024-04-24T01:58:16.086 INFO:tasks.rados.rados.0.smithi079.stdout:694: done (2 left) 2024-04-24T01:58:16.086 INFO:tasks.rados.rados.0.smithi079.stdout:695: read oid 23 snap 53 2024-04-24T01:58:16.086 INFO:tasks.rados.rados.0.smithi079.stdout:695: expect (ObjNum 206 snap 44 seq_num 206) 2024-04-24T01:58:16.086 INFO:tasks.rados.rados.0.smithi079.stdout:696: read oid 16 snap -1 2024-04-24T01:58:16.086 INFO:tasks.rados.rados.0.smithi079.stdout:696: expect deleted 2024-04-24T01:58:16.086 INFO:tasks.rados.rados.0.smithi079.stdout:697: read oid 26 snap -1 2024-04-24T01:58:16.086 INFO:tasks.rados.rados.0.smithi079.stdout:697: expect deleted 2024-04-24T01:58:16.086 INFO:tasks.rados.rados.0.smithi079.stdout:698: read oid 47 snap -1 2024-04-24T01:58:16.086 INFO:tasks.rados.rados.0.smithi079.stdout:698: expect (ObjNum 323 snap 79 seq_num 323) 2024-04-24T01:58:16.086 INFO:tasks.rados.rados.0.smithi079.stdout:699: snap_remove snap 81 2024-04-24T01:58:17.040 INFO:tasks.rados.rados.0.smithi079.stdout:699: done (6 left) 2024-04-24T01:58:17.040 INFO:tasks.rados.rados.0.smithi079.stdout:700: read oid 32 snap 79 2024-04-24T01:58:17.040 INFO:tasks.rados.rados.0.smithi079.stdout:700: expect (ObjNum 313 snap 79 seq_num 313) 2024-04-24T01:58:17.040 INFO:tasks.rados.rados.0.smithi079.stdout:701: delete oid 46 current snap is 82 2024-04-24T01:58:17.040 INFO:tasks.rados.rados.0.smithi079.stdout:698: expect (ObjNum 323 snap 79 seq_num 323) 2024-04-24T01:58:17.282 INFO:tasks.rados.rados.0.smithi079.stdout:693: expect (ObjNum 290 snap 76 seq_num 290) 2024-04-24T01:58:17.422 INFO:tasks.rados.rados.0.smithi079.stdout:695: expect (ObjNum 206 snap 44 seq_num 206) 2024-04-24T01:58:17.552 INFO:tasks.rados.rados.0.smithi079.stdout:700: expect (ObjNum 313 snap 79 seq_num 313) 2024-04-24T01:58:17.623 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T01:58:17.676 INFO:tasks.rados.rados.0.smithi079.stdout:692: done (7 left) 2024-04-24T01:58:17.676 INFO:tasks.rados.rados.0.smithi079.stdout:693: done (6 left) 2024-04-24T01:58:17.676 INFO:tasks.rados.rados.0.smithi079.stdout:695: done (5 left) 2024-04-24T01:58:17.677 INFO:tasks.rados.rados.0.smithi079.stdout:696: done (4 left) 2024-04-24T01:58:17.677 INFO:tasks.rados.rados.0.smithi079.stdout:697: done (3 left) 2024-04-24T01:58:17.677 INFO:tasks.rados.rados.0.smithi079.stdout:698: done (2 left) 2024-04-24T01:58:17.677 INFO:tasks.rados.rados.0.smithi079.stdout:700: done (1 left) 2024-04-24T01:58:17.677 INFO:tasks.rados.rados.0.smithi079.stdout:701: done (0 left) 2024-04-24T01:58:17.677 INFO:tasks.rados.rados.0.smithi079.stdout:702: snap_remove snap 53 2024-04-24T01:58:17.727 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:58:17.725+0000 7f037b35d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:17.731 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:58:17.730+0000 7f037b35d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:18.134 INFO:tasks.rados.rados.0.smithi079.stdout:702: done (0 left) 2024-04-24T01:58:18.134 INFO:tasks.rados.rados.0.smithi079.stdout:703: write oid 13 current snap is 82 2024-04-24T01:58:18.134 INFO:tasks.rados.rados.0.smithi079.stdout:703: seq_num 341 ranges {531187=779441,2088242=738446,3431711=299476} 2024-04-24T01:58:18.144 INFO:tasks.rados.rados.0.smithi079.stdout:703: writing smithi07929762-13 from 531187 to 1310628 tid 1 2024-04-24T01:58:18.154 INFO:tasks.rados.rados.0.smithi079.stdout:703: writing smithi07929762-13 from 2088242 to 2826688 tid 2 2024-04-24T01:58:18.161 INFO:tasks.rados.rados.0.smithi079.stdout:703: writing smithi07929762-13 from 3431711 to 3731187 tid 3 2024-04-24T01:58:18.161 INFO:tasks.rados.rados.0.smithi079.stdout:704: read oid 49 snap 79 2024-04-24T01:58:18.161 INFO:tasks.rados.rados.0.smithi079.stdout:704: expect deleted 2024-04-24T01:58:18.161 INFO:tasks.rados.rados.0.smithi079.stdout:705: read oid 16 snap -1 2024-04-24T01:58:18.161 INFO:tasks.rados.rados.0.smithi079.stdout:705: expect deleted 2024-04-24T01:58:18.162 INFO:tasks.rados.rados.0.smithi079.stdout:706: read oid 37 snap 79 2024-04-24T01:58:18.162 INFO:tasks.rados.rados.0.smithi079.stdout:706: expect deleted 2024-04-24T01:58:18.162 INFO:tasks.rados.rados.0.smithi079.stdout:707: read oid 10 snap -1 2024-04-24T01:58:18.162 INFO:tasks.rados.rados.0.smithi079.stdout:707: expect (ObjNum 338 snap 81 seq_num 338) 2024-04-24T01:58:18.162 INFO:tasks.rados.rados.0.smithi079.stdout:708: delete oid 3 current snap is 82 2024-04-24T01:58:18.356 INFO:tasks.rados.rados.0.smithi079.stdout:704: done (5 left) 2024-04-24T01:58:18.356 INFO:tasks.rados.rados.0.smithi079.stdout:705: done (4 left) 2024-04-24T01:58:18.356 INFO:tasks.rados.rados.0.smithi079.stdout:706: done (3 left) 2024-04-24T01:58:18.356 INFO:tasks.rados.rados.0.smithi079.stdout:708: done (2 left) 2024-04-24T01:58:18.356 INFO:tasks.rados.rados.0.smithi079.stdout:709: write oid 50 current snap is 82 2024-04-24T01:58:18.356 INFO:tasks.rados.rados.0.smithi079.stdout:709: seq_num 343 ranges {564801=646292,1921559=742115,3202910=561891} 2024-04-24T01:58:18.366 INFO:tasks.rados.rados.0.smithi079.stdout:709: writing smithi07929762-50 from 564801 to 1211093 tid 1 2024-04-24T01:58:18.378 INFO:tasks.rados.rados.0.smithi079.stdout:709: writing smithi07929762-50 from 1921559 to 2663674 tid 2 2024-04-24T01:58:18.385 INFO:tasks.rados.rados.0.smithi079.stdout:709: writing smithi07929762-50 from 3202910 to 3764801 tid 3 2024-04-24T01:58:18.385 INFO:tasks.rados.rados.0.smithi079.stdout:710: snap_remove snap 79 2024-04-24T01:58:19.137 INFO:tasks.rados.rados.0.smithi079.stdout:710: done (3 left) 2024-04-24T01:58:19.138 INFO:tasks.rados.rados.0.smithi079.stdout:711: write (excl) oid 25 current snap is 82 2024-04-24T01:58:19.138 INFO:tasks.rados.rados.0.smithi079.stdout:711: seq_num 344 ranges {581608=721541,1863688=485773,3113980=604043} 2024-04-24T01:58:19.144 INFO:tasks.rados.rados.0.smithi079.stdout:711: writing smithi07929762-25 from 581608 to 1303149 tid 1 2024-04-24T01:58:19.150 INFO:tasks.rados.rados.0.smithi079.stdout:711: writing smithi07929762-25 from 1863688 to 2349461 tid 2 2024-04-24T01:58:19.158 INFO:tasks.rados.rados.0.smithi079.stdout:711: writing smithi07929762-25 from 3113980 to 3718023 tid 3 2024-04-24T01:58:19.158 INFO:tasks.rados.rados.0.smithi079.stdout:712: write oid 47 current snap is 82 2024-04-24T01:58:19.158 INFO:tasks.rados.rados.0.smithi079.stdout:707: expect (ObjNum 338 snap 81 seq_num 338) 2024-04-24T01:58:19.385 INFO:tasks.rados.rados.0.smithi079.stdout:709: finishing write tid 1 to smithi07929762-50 2024-04-24T01:58:19.385 INFO:tasks.rados.rados.0.smithi079.stdout:703: finishing write tid 1 to smithi07929762-13 2024-04-24T01:58:19.385 INFO:tasks.rados.rados.0.smithi079.stdout:709: finishing write tid 2 to smithi07929762-50 2024-04-24T01:58:19.385 INFO:tasks.rados.rados.0.smithi079.stdout:703: finishing write tid 2 to smithi07929762-13 2024-04-24T01:58:19.385 INFO:tasks.rados.rados.0.smithi079.stdout:709: finishing write tid 3 to smithi07929762-50 2024-04-24T01:58:19.385 INFO:tasks.rados.rados.0.smithi079.stdout:703: finishing write tid 3 to smithi07929762-13 2024-04-24T01:58:19.385 INFO:tasks.rados.rados.0.smithi079.stdout:709: finishing write tid 4 to smithi07929762-50 2024-04-24T01:58:19.386 INFO:tasks.rados.rados.0.smithi079.stdout:712: seq_num 345 ranges {598415=796790,1922170=745784,3141403=415315} 2024-04-24T01:58:19.392 INFO:tasks.rados.rados.0.smithi079.stdout:712: writing smithi07929762-47 from 598415 to 1395205 tid 1 2024-04-24T01:58:19.402 INFO:tasks.rados.rados.0.smithi079.stdout:712: writing smithi07929762-47 from 1922170 to 2667954 tid 2 2024-04-24T01:58:19.407 INFO:tasks.rados.rados.0.smithi079.stdout:712: writing smithi07929762-47 from 3141403 to 3556718 tid 3 2024-04-24T01:58:19.407 INFO:tasks.rados.rados.0.smithi079.stdout:707: done (4 left) 2024-04-24T01:58:19.408 INFO:tasks.rados.rados.0.smithi079.stdout:713: read oid 48 snap -1 2024-04-24T01:58:19.408 INFO:tasks.rados.rados.0.smithi079.stdout:713: expect (ObjNum 284 snap 75 seq_num 284) 2024-04-24T01:58:19.410 INFO:tasks.rados.rados.0.smithi079.stdout:714: write (excl) oid 35 current snap is 82 2024-04-24T01:58:19.411 INFO:tasks.rados.rados.0.smithi079.stdout:714: seq_num 346 ranges {615222=472039,1580652=489442,2768826=626587} 2024-04-24T01:58:19.415 INFO:tasks.rados.rados.0.smithi079.stdout:714: writing smithi07929762-35 from 615222 to 1087261 tid 1 2024-04-24T01:58:19.421 INFO:tasks.rados.rados.0.smithi079.stdout:714: writing smithi07929762-35 from 1580652 to 2070094 tid 2 2024-04-24T01:58:19.431 INFO:tasks.rados.rados.0.smithi079.stdout:714: writing smithi07929762-35 from 2768826 to 3395413 tid 3 2024-04-24T01:58:19.433 INFO:tasks.rados.rados.0.smithi079.stdout:715: write (excl) oid 33 current snap is 82 2024-04-24T01:58:19.433 INFO:tasks.rados.rados.0.smithi079.stdout:709: finishing write tid 5 to smithi07929762-50 2024-04-24T01:58:19.433 INFO:tasks.rados.rados.0.smithi079.stdout:709: oid 50 updating version 0 to 187 2024-04-24T01:58:19.433 INFO:tasks.rados.rados.0.smithi079.stdout:709: oid 50 version 187 is already newer than 187 2024-04-24T01:58:19.433 INFO:tasks.rados.rados.0.smithi079.stdout:709: oid 50 version 187 is already newer than 187 2024-04-24T01:58:19.433 INFO:tasks.rados.rados.0.smithi079.stdout:709: oid 50 version 187 is already newer than 187 2024-04-24T01:58:19.434 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 50 v 187 (ObjNum 343 snap 82 seq_num 343) dirty exists 2024-04-24T01:58:19.434 INFO:tasks.rados.rados.0.smithi079.stdout:709: left oid 50 (ObjNum 343 snap 82 seq_num 343) 2024-04-24T01:58:19.434 INFO:tasks.rados.rados.0.smithi079.stdout:703: finishing write tid 4 to smithi07929762-13 2024-04-24T01:58:19.434 INFO:tasks.rados.rados.0.smithi079.stdout:703: finishing write tid 5 to smithi07929762-13 2024-04-24T01:58:19.434 INFO:tasks.rados.rados.0.smithi079.stdout:703: oid 13 updating version 0 to 275 2024-04-24T01:58:19.434 INFO:tasks.rados.rados.0.smithi079.stdout:703: oid 13 version 275 is already newer than 275 2024-04-24T01:58:19.434 INFO:tasks.rados.rados.0.smithi079.stdout:703: oid 13 version 275 is already newer than 275 2024-04-24T01:58:19.434 INFO:tasks.rados.rados.0.smithi079.stdout:703: oid 13 version 275 is already newer than 275 2024-04-24T01:58:19.434 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 13 v 275 (ObjNum 341 snap 82 seq_num 341) dirty exists 2024-04-24T01:58:19.435 INFO:tasks.rados.rados.0.smithi079.stdout:703: left oid 13 (ObjNum 341 snap 82 seq_num 341) 2024-04-24T01:58:19.435 INFO:tasks.rados.rados.0.smithi079.stdout:711: finishing write tid 1 to smithi07929762-25 2024-04-24T01:58:19.435 INFO:tasks.rados.rados.0.smithi079.stdout:711: finishing write tid 2 to smithi07929762-25 2024-04-24T01:58:19.435 INFO:tasks.rados.rados.0.smithi079.stdout:715: seq_num 347 ranges {632029=547288,1639134=749453,2796249=437859,3653524=178505} 2024-04-24T01:58:19.441 INFO:tasks.rados.rados.0.smithi079.stdout:715: writing smithi07929762-33 from 632029 to 1179317 tid 1 2024-04-24T01:58:19.451 INFO:tasks.rados.rados.0.smithi079.stdout:715: writing smithi07929762-33 from 1639134 to 2388587 tid 2 2024-04-24T01:58:19.458 INFO:tasks.rados.rados.0.smithi079.stdout:715: writing smithi07929762-33 from 2796249 to 3234108 tid 3 2024-04-24T01:58:19.462 INFO:tasks.rados.rados.0.smithi079.stdout:715: writing smithi07929762-33 from 3653524 to 3832029 tid 4 2024-04-24T01:58:19.462 INFO:tasks.rados.rados.0.smithi079.stdout:703: done (6 left) 2024-04-24T01:58:19.463 INFO:tasks.rados.rados.0.smithi079.stdout:709: done (5 left) 2024-04-24T01:58:19.463 INFO:tasks.rados.rados.0.smithi079.stdout:716: write oid 44 current snap is 82 2024-04-24T01:58:19.463 INFO:tasks.rados.rados.0.smithi079.stdout:716: seq_num 348 ranges {648836=622537,1981263=493111,3107319=741517} 2024-04-24T01:58:19.472 INFO:tasks.rados.rados.0.smithi079.stdout:716: writing smithi07929762-44 from 648836 to 1271373 tid 1 2024-04-24T01:58:19.481 INFO:tasks.rados.rados.0.smithi079.stdout:716: writing smithi07929762-44 from 1981263 to 2474374 tid 2 2024-04-24T01:58:19.492 INFO:tasks.rados.rados.0.smithi079.stdout:716: writing smithi07929762-44 from 3107319 to 3848836 tid 3 2024-04-24T01:58:19.492 INFO:tasks.rados.rados.0.smithi079.stdout:717: delete oid 21 current snap is 82 2024-04-24T01:58:19.493 INFO:tasks.rados.rados.0.smithi079.stdout:711: finishing write tid 3 to smithi07929762-25 2024-04-24T01:58:19.493 INFO:tasks.rados.rados.0.smithi079.stdout:711: finishing write tid 4 to smithi07929762-25 2024-04-24T01:58:19.493 INFO:tasks.rados.rados.0.smithi079.stdout:711: finishing write tid 5 to smithi07929762-25 2024-04-24T01:58:19.493 INFO:tasks.rados.rados.0.smithi079.stdout:711: oid 25 updating version 0 to 285 2024-04-24T01:58:19.493 INFO:tasks.rados.rados.0.smithi079.stdout:711: oid 25 version 285 is already newer than 285 2024-04-24T01:58:19.493 INFO:tasks.rados.rados.0.smithi079.stdout:711: oid 25 version 285 is already newer than 285 2024-04-24T01:58:19.493 INFO:tasks.rados.rados.0.smithi079.stdout:711: oid 25 version 285 is already newer than 282 2024-04-24T01:58:19.493 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 25 v 285 (ObjNum 344 snap 82 seq_num 344) dirty exists 2024-04-24T01:58:19.493 INFO:tasks.rados.rados.0.smithi079.stdout:711: left oid 25 (ObjNum 344 snap 82 seq_num 344) 2024-04-24T01:58:19.493 INFO:tasks.rados.rados.0.smithi079.stdout:712: finishing write tid 1 to smithi07929762-47 2024-04-24T01:58:19.493 INFO:tasks.rados.rados.0.smithi079.stdout:713: expect (ObjNum 284 snap 75 seq_num 284) 2024-04-24T01:58:19.612 INFO:tasks.rados.rados.0.smithi079.stdout:712: finishing write tid 2 to smithi07929762-47 2024-04-24T01:58:19.613 INFO:tasks.rados.rados.0.smithi079.stdout:714: finishing write tid 1 to smithi07929762-35 2024-04-24T01:58:19.613 INFO:tasks.rados.rados.0.smithi079.stdout:712: finishing write tid 3 to smithi07929762-47 2024-04-24T01:58:19.613 INFO:tasks.rados.rados.0.smithi079.stdout:712: finishing write tid 4 to smithi07929762-47 2024-04-24T01:58:19.613 INFO:tasks.rados.rados.0.smithi079.stdout:712: finishing write tid 5 to smithi07929762-47 2024-04-24T01:58:19.613 INFO:tasks.rados.rados.0.smithi079.stdout:712: oid 47 updating version 0 to 289 2024-04-24T01:58:19.613 INFO:tasks.rados.rados.0.smithi079.stdout:712: oid 47 version 289 is already newer than 289 2024-04-24T01:58:19.613 INFO:tasks.rados.rados.0.smithi079.stdout:712: oid 47 version 289 is already newer than 289 2024-04-24T01:58:19.613 INFO:tasks.rados.rados.0.smithi079.stdout:712: oid 47 version 289 is already newer than 289 2024-04-24T01:58:19.613 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 47 v 289 (ObjNum 345 snap 82 seq_num 345) dirty exists 2024-04-24T01:58:19.613 INFO:tasks.rados.rados.0.smithi079.stdout:712: left oid 47 (ObjNum 345 snap 82 seq_num 345) 2024-04-24T01:58:19.613 INFO:tasks.rados.rados.0.smithi079.stdout:715: finishing write tid 1 to smithi07929762-33 2024-04-24T01:58:19.613 INFO:tasks.rados.rados.0.smithi079.stdout:711: done (6 left) 2024-04-24T01:58:19.613 INFO:tasks.rados.rados.0.smithi079.stdout:712: done (5 left) 2024-04-24T01:58:19.614 INFO:tasks.rados.rados.0.smithi079.stdout:713: done (4 left) 2024-04-24T01:58:19.614 INFO:tasks.rados.rados.0.smithi079.stdout:717: done (3 left) 2024-04-24T01:58:19.614 INFO:tasks.rados.rados.0.smithi079.stdout:718: read oid 4 snap -1 2024-04-24T01:58:19.614 INFO:tasks.rados.rados.0.smithi079.stdout:718: expect deleted 2024-04-24T01:58:19.614 INFO:tasks.rados.rados.0.smithi079.stdout:719: read oid 12 snap -1 2024-04-24T01:58:19.614 INFO:tasks.rados.rados.0.smithi079.stdout:719: expect deleted 2024-04-24T01:58:19.614 INFO:tasks.rados.rados.0.smithi079.stdout:720: snap_create 2024-04-24T01:58:19.614 INFO:tasks.rados.rados.0.smithi079.stdout:714: finishing write tid 2 to smithi07929762-35 2024-04-24T01:58:19.614 INFO:tasks.rados.rados.0.smithi079.stdout:716: finishing write tid 1 to smithi07929762-44 2024-04-24T01:58:19.614 INFO:tasks.rados.rados.0.smithi079.stdout:714: finishing write tid 3 to smithi07929762-35 2024-04-24T01:58:19.614 INFO:tasks.rados.rados.0.smithi079.stdout:714: finishing write tid 4 to smithi07929762-35 2024-04-24T01:58:19.614 INFO:tasks.rados.rados.0.smithi079.stdout:716: finishing write tid 2 to smithi07929762-44 2024-04-24T01:58:19.614 INFO:tasks.rados.rados.0.smithi079.stdout:714: finishing write tid 5 to smithi07929762-35 2024-04-24T01:58:19.614 INFO:tasks.rados.rados.0.smithi079.stdout:714: oid 35 updating version 0 to 229 2024-04-24T01:58:19.614 INFO:tasks.rados.rados.0.smithi079.stdout:714: oid 35 version 229 is already newer than 226 2024-04-24T01:58:19.615 INFO:tasks.rados.rados.0.smithi079.stdout:714: oid 35 version 229 is already newer than 229 2024-04-24T01:58:19.615 INFO:tasks.rados.rados.0.smithi079.stdout:714: oid 35 version 229 is already newer than 229 2024-04-24T01:58:19.615 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 35 v 229 (ObjNum 346 snap 82 seq_num 346) dirty exists 2024-04-24T01:58:19.615 INFO:tasks.rados.rados.0.smithi079.stdout:714: left oid 35 (ObjNum 346 snap 82 seq_num 346) 2024-04-24T01:58:19.615 INFO:tasks.rados.rados.0.smithi079.stdout:716: finishing write tid 3 to smithi07929762-44 2024-04-24T01:58:19.615 INFO:tasks.rados.rados.0.smithi079.stdout:716: finishing write tid 4 to smithi07929762-44 2024-04-24T01:58:19.615 INFO:tasks.rados.rados.0.smithi079.stdout:716: finishing write tid 5 to smithi07929762-44 2024-04-24T01:58:19.615 INFO:tasks.rados.rados.0.smithi079.stdout:716: oid 44 updating version 0 to 194 2024-04-24T01:58:19.615 INFO:tasks.rados.rados.0.smithi079.stdout:716: oid 44 version 194 is already newer than 194 2024-04-24T01:58:19.615 INFO:tasks.rados.rados.0.smithi079.stdout:716: oid 44 version 194 is already newer than 194 2024-04-24T01:58:19.615 INFO:tasks.rados.rados.0.smithi079.stdout:716: oid 44 version 194 is already newer than 194 2024-04-24T01:58:19.615 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 44 v 194 (ObjNum 348 snap 82 seq_num 348) dirty exists 2024-04-24T01:58:19.615 INFO:tasks.rados.rados.0.smithi079.stdout:716: left oid 44 (ObjNum 348 snap 82 seq_num 348) 2024-04-24T01:58:19.615 INFO:tasks.rados.rados.0.smithi079.stdout:715: finishing write tid 2 to smithi07929762-33 2024-04-24T01:58:19.615 INFO:tasks.rados.rados.0.smithi079.stdout:715: finishing write tid 3 to smithi07929762-33 2024-04-24T01:58:19.615 INFO:tasks.rados.rados.0.smithi079.stdout:715: finishing write tid 4 to smithi07929762-33 2024-04-24T01:58:19.616 INFO:tasks.rados.rados.0.smithi079.stdout:715: finishing write tid 5 to smithi07929762-33 2024-04-24T01:58:19.616 INFO:tasks.rados.rados.0.smithi079.stdout:715: finishing write tid 6 to smithi07929762-33 2024-04-24T01:58:19.616 INFO:tasks.rados.rados.0.smithi079.stdout:715: oid 33 updating version 0 to 250 2024-04-24T01:58:19.616 INFO:tasks.rados.rados.0.smithi079.stdout:715: oid 33 version 250 is already newer than 246 2024-04-24T01:58:19.616 INFO:tasks.rados.rados.0.smithi079.stdout:715: oid 33 version 250 is already newer than 250 2024-04-24T01:58:19.616 INFO:tasks.rados.rados.0.smithi079.stdout:715: oid 33 version 250 is already newer than 250 2024-04-24T01:58:19.616 INFO:tasks.rados.rados.0.smithi079.stdout:715: oid 33 version 250 is already newer than 250 2024-04-24T01:58:19.616 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 33 v 250 (ObjNum 347 snap 82 seq_num 347) dirty exists 2024-04-24T01:58:19.616 INFO:tasks.rados.rados.0.smithi079.stdout:715: left oid 33 (ObjNum 347 snap 82 seq_num 347) 2024-04-24T01:58:19.826 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T01:58:19.932 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:58:19.930+0000 7feda175d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:19.936 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:58:19.935+0000 7feda175d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:20.141 INFO:tasks.rados.rados.0.smithi079.stdout:714: done (5 left) 2024-04-24T01:58:20.141 INFO:tasks.rados.rados.0.smithi079.stdout:715: done (4 left) 2024-04-24T01:58:20.141 INFO:tasks.rados.rados.0.smithi079.stdout:716: done (3 left) 2024-04-24T01:58:20.141 INFO:tasks.rados.rados.0.smithi079.stdout:718: done (2 left) 2024-04-24T01:58:20.141 INFO:tasks.rados.rados.0.smithi079.stdout:719: done (1 left) 2024-04-24T01:58:20.141 INFO:tasks.rados.rados.0.smithi079.stdout:720: done (0 left) 2024-04-24T01:58:20.141 INFO:tasks.rados.rados.0.smithi079.stdout:721: read oid 15 snap 82 2024-04-24T01:58:20.141 INFO:tasks.rados.rados.0.smithi079.stdout:721: expect (ObjNum 197 snap 40 seq_num 197) 2024-04-24T01:58:20.141 INFO:tasks.rados.rados.0.smithi079.stdout:722: read oid 48 snap 82 2024-04-24T01:58:20.142 INFO:tasks.rados.rados.0.smithi079.stdout:722: expect (ObjNum 284 snap 75 seq_num 284) 2024-04-24T01:58:20.142 INFO:tasks.rados.rados.0.smithi079.stdout:723: snap_create 2024-04-24T01:58:20.185 INFO:tasks.rados.rados.0.smithi079.stdout:722: expect (ObjNum 284 snap 75 seq_num 284) 2024-04-24T01:58:20.291 INFO:tasks.rados.rados.0.smithi079.stdout:721: expect (ObjNum 197 snap 40 seq_num 197) 2024-04-24T01:58:20.327 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-04-24T01:58:20.328 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-04-24T01:58:20.328 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T01:58:20.435 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:58:20.431+0000 7f4e65e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:20.439 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:58:20.438+0000 7f4e65e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:20.631 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:58:20.631 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-24T01:58:20.644 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":268,"stamp":"2024-04-24T01:58:20.139969+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":95718714,"num_objects":57,"num_object_clones":19,"num_object_copies":171,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":1457,"num_read_kb":955194,"num_write":1312,"num_write_kb":323494,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2699,"ondisk_log_size":2699,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":454676,"kb_used_data":320712,"kb_used_omap":0,"kb_used_meta":123904,"kb_avail":377032684,"statfs":{"total":386547056640,"available":386081468416,"internally_reserved":0,"allocated":328409088,"data_stored":341610715,"data_compressed":89889,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":126877696},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-10806703,"num_objects":-43,"num_object_clones":-36,"num_object_copies":-129,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":-5,"num_read":34,"num_read_kb":32221,"num_write":12,"num_write_kb":3436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.104967"},"pg_stats":[{"pgid":"3.f","version":"182'127","reported_seq":643,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-04-24T01:58:18.229255+0000","last_change":"2024-04-24T01:58:18.456803+0000","last_active":"2024-04-24T01:58:18.229255+0000","last_peered":"2024-04-24T01:58:18.229255+0000","last_clean":"2024-04-24T01:58:18.229255+0000","last_became_active":"2024-04-24T01:55:30.386726+0000","last_became_peered":"2024-04-24T01:55:30.386726+0000","last_unstale":"2024-04-24T01:58:18.229255+0000","last_undegraded":"2024-04-24T01:58:18.229255+0000","last_fullsized":"2024-04-24T01:58:18.229255+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":10,"snaptrim_duration":0,"stat_sum":{"num_bytes":6033779,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":80,"num_read_kb":69693,"num_write":53,"num_write_kb":15604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"182'215","reported_seq":714,"reported_epoch":182,"state":"active+clean+snaptrim","last_fresh":"2024-04-24T01:58:18.371499+0000","last_change":"2024-04-24T01:58:18.194705+0000","last_active":"2024-04-24T01:58:18.371499+0000","last_peered":"2024-04-24T01:58:18.371499+0000","last_clean":"2024-04-24T01:58:18.371499+0000","last_became_active":"2024-04-24T01:55:30.321866+0000","last_became_peered":"2024-04-24T01:55:30.321866+0000","last_unstale":"2024-04-24T01:58:18.371499+0000","last_undegraded":"2024-04-24T01:58:18.371499+0000","last_fullsized":"2024-04-24T01:58:18.371499+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":215,"log_dups_size":0,"ondisk_log_size":215,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":11,"snaptrim_duration":0,"stat_sum":{"num_bytes":12823873,"num_objects":8,"num_object_clones":4,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":125,"num_read_kb":82519,"num_write":100,"num_write_kb":27747,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"180'181","reported_seq":669,"reported_epoch":182,"state":"active+clean+snaptrim","last_fresh":"2024-04-24T01:58:18.182493+0000","last_change":"2024-04-24T01:58:18.182493+0000","last_active":"2024-04-24T01:58:18.182493+0000","last_peered":"2024-04-24T01:58:18.182493+0000","last_clean":"2024-04-24T01:58:18.182493+0000","last_became_active":"2024-04-24T01:55:30.320186+0000","last_became_peered":"2024-04-24T01:55:30.320186+0000","last_unstale":"2024-04-24T01:58:18.182493+0000","last_undegraded":"2024-04-24T01:58:18.182493+0000","last_fullsized":"2024-04-24T01:58:18.182493+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":181,"log_dups_size":0,"ondisk_log_size":181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":14,"snaptrim_duration":0,"stat_sum":{"num_bytes":6353112,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":64,"num_read_kb":35668,"num_write":95,"num_write_kb":25777,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"182'189","reported_seq":695,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-04-24T01:58:18.655938+0000","last_change":"2024-04-24T01:58:18.674348+0000","last_active":"2024-04-24T01:58:18.655938+0000","last_peered":"2024-04-24T01:58:18.655938+0000","last_clean":"2024-04-24T01:58:18.655938+0000","last_became_active":"2024-04-24T01:55:30.288081+0000","last_became_peered":"2024-04-24T01:55:30.288081+0000","last_unstale":"2024-04-24T01:58:18.655938+0000","last_undegraded":"2024-04-24T01:58:18.655938+0000","last_fullsized":"2024-04-24T01:58:18.655938+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":189,"log_dups_size":0,"ondisk_log_size":189,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":15,"snaptrim_duration":0,"stat_sum":{"num_bytes":10000264,"num_objects":6,"num_object_clones":2,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":107,"num_read_kb":77554,"num_write":80,"num_write_kb":23066,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"182'61","reported_seq":583,"reported_epoch":183,"state":"active+clean+snaptrim","last_fresh":"2024-04-24T01:58:19.154528+0000","last_change":"2024-04-24T01:58:19.154528+0000","last_active":"2024-04-24T01:58:19.154528+0000","last_peered":"2024-04-24T01:58:19.154528+0000","last_clean":"2024-04-24T01:58:19.154528+0000","last_became_active":"2024-04-24T01:55:30.333815+0000","last_became_peered":"2024-04-24T01:55:30.333815+0000","last_unstale":"2024-04-24T01:58:19.154528+0000","last_undegraded":"2024-04-24T01:58:19.154528+0000","last_fullsized":"2024-04-24T01:58:19.154528+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":5,"snaptrim_duration":0,"stat_sum":{"num_bytes":3798382,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":32,"num_read_kb":17556,"num_write":36,"num_write_kb":10196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"182'397","reported_seq":857,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-04-24T01:58:18.555746+0000","last_change":"2024-04-24T01:58:18.586814+0000","last_active":"2024-04-24T01:58:18.555746+0000","last_peered":"2024-04-24T01:58:18.555746+0000","last_clean":"2024-04-24T01:58:18.555746+0000","last_became_active":"2024-04-24T01:55:30.315075+0000","last_became_peered":"2024-04-24T01:55:30.315075+0000","last_unstale":"2024-04-24T01:58:18.555746+0000","last_undegraded":"2024-04-24T01:58:18.555746+0000","last_fullsized":"2024-04-24T01:58:18.555746+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":397,"log_dups_size":0,"ondisk_log_size":397,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":34,"snaptrim_duration":0,"stat_sum":{"num_bytes":7193462,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":169,"num_read_kb":102269,"num_write":162,"num_write_kb":43146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"182'64","reported_seq":584,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-04-24T01:58:18.171861+0000","last_change":"2024-04-24T01:58:18.230747+0000","last_active":"2024-04-24T01:58:18.171861+0000","last_peered":"2024-04-24T01:58:18.171861+0000","last_clean":"2024-04-24T01:58:18.171861+0000","last_became_active":"2024-04-24T01:55:30.350300+0000","last_became_peered":"2024-04-24T01:55:30.350300+0000","last_unstale":"2024-04-24T01:58:18.171861+0000","last_undegraded":"2024-04-24T01:58:18.171861+0000","last_fullsized":"2024-04-24T01:58:18.171861+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":5,"snaptrim_duration":0,"stat_sum":{"num_bytes":3596731,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":33,"num_read_kb":16964,"num_write":41,"num_write_kb":10355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"163'8","reported_seq":519,"reported_epoch":182,"state":"active+clean+snaptrim","last_fresh":"2024-04-24T01:58:18.191943+0000","last_change":"2024-04-24T01:58:18.191943+0000","last_active":"2024-04-24T01:58:18.191943+0000","last_peered":"2024-04-24T01:58:18.191943+0000","last_clean":"2024-04-24T01:58:18.191943+0000","last_became_active":"2024-04-24T01:55:30.376813+0000","last_became_peered":"2024-04-24T01:55:30.376813+0000","last_unstale":"2024-04-24T01:58:18.191943+0000","last_undegraded":"2024-04-24T01:58:18.191943+0000","last_fullsized":"2024-04-24T01:58:18.191943+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'97","reported_seq":585,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-04-24T01:58:19.153558+0000","last_change":"2024-04-24T01:55:16.196470+0000","last_active":"2024-04-24T01:58:19.153558+0000","last_peered":"2024-04-24T01:58:19.153558+0000","last_clean":"2024-04-24T01:58:19.153558+0000","last_became_active":"2024-04-24T01:55:16.190464+0000","last_became_peered":"2024-04-24T01:55:16.190464+0000","last_unstale":"2024-04-24T01:58:19.153558+0000","last_undegraded":"2024-04-24T01:58:19.153558+0000","last_fullsized":"2024-04-24T01:58:19.153558+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:15.127115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:15.127115+0000","last_clean_scrub_stamp":"2024-04-24T01:55:15.127115+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":440,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-04-24T01:58:18.159523+0000","last_change":"2024-04-24T01:55:18.236148+0000","last_active":"2024-04-24T01:58:18.159523+0000","last_peered":"2024-04-24T01:58:18.159523+0000","last_clean":"2024-04-24T01:58:18.159523+0000","last_became_active":"2024-04-24T01:55:18.230154+0000","last_became_peered":"2024-04-24T01:55:18.230154+0000","last_unstale":"2024-04-24T01:58:18.159523+0000","last_undegraded":"2024-04-24T01:58:18.159523+0000","last_fullsized":"2024-04-24T01:58:18.159523+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"180'267","reported_seq":771,"reported_epoch":182,"state":"active+clean+snaptrim","last_fresh":"2024-04-24T01:58:18.190528+0000","last_change":"2024-04-24T01:58:18.190528+0000","last_active":"2024-04-24T01:58:18.190528+0000","last_peered":"2024-04-24T01:58:18.190528+0000","last_clean":"2024-04-24T01:58:18.190528+0000","last_became_active":"2024-04-24T01:55:30.316222+0000","last_became_peered":"2024-04-24T01:55:30.316222+0000","last_unstale":"2024-04-24T01:58:18.190528+0000","last_undegraded":"2024-04-24T01:58:18.190528+0000","last_fullsized":"2024-04-24T01:58:18.190528+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":267,"log_dups_size":0,"ondisk_log_size":267,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":16,"snaptrim_duration":0,"stat_sum":{"num_bytes":9008552,"num_objects":9,"num_object_clones":5,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":153,"num_read_kb":107362,"num_write":127,"num_write_kb":34941,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":449,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-04-24T01:58:18.158819+0000","last_change":"2024-04-24T01:55:18.265905+0000","last_active":"2024-04-24T01:58:18.158819+0000","last_peered":"2024-04-24T01:58:18.158819+0000","last_clean":"2024-04-24T01:58:18.158819+0000","last_became_active":"2024-04-24T01:55:18.257020+0000","last_became_peered":"2024-04-24T01:55:18.257020+0000","last_unstale":"2024-04-24T01:58:18.158819+0000","last_undegraded":"2024-04-24T01:58:18.158819+0000","last_fullsized":"2024-04-24T01:58:18.158819+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"182'225","reported_seq":715,"reported_epoch":183,"state":"active+clean+snaptrim","last_fresh":"2024-04-24T01:58:19.149166+0000","last_change":"2024-04-24T01:58:19.149166+0000","last_active":"2024-04-24T01:58:19.149166+0000","last_peered":"2024-04-24T01:58:19.149166+0000","last_clean":"2024-04-24T01:58:19.149166+0000","last_became_active":"2024-04-24T01:55:30.344129+0000","last_became_peered":"2024-04-24T01:55:30.344129+0000","last_unstale":"2024-04-24T01:58:19.149166+0000","last_undegraded":"2024-04-24T01:58:19.149166+0000","last_fullsized":"2024-04-24T01:58:19.149166+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":225,"log_dups_size":0,"ondisk_log_size":225,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":15,"snaptrim_duration":0,"stat_sum":{"num_bytes":6151428,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":94,"num_read_kb":60231,"num_write":91,"num_write_kb":26485,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":443,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-04-24T01:58:18.181150+0000","last_change":"2024-04-24T01:55:18.215251+0000","last_active":"2024-04-24T01:58:18.181150+0000","last_peered":"2024-04-24T01:58:18.181150+0000","last_clean":"2024-04-24T01:58:18.181150+0000","last_became_active":"2024-04-24T01:55:18.206605+0000","last_became_peered":"2024-04-24T01:55:18.206605+0000","last_unstale":"2024-04-24T01:58:18.181150+0000","last_undegraded":"2024-04-24T01:58:18.181150+0000","last_fullsized":"2024-04-24T01:58:18.181150+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"182'118","reported_seq":636,"reported_epoch":183,"state":"active+clean+snaptrim","last_fresh":"2024-04-24T01:58:19.156759+0000","last_change":"2024-04-24T01:58:19.156759+0000","last_active":"2024-04-24T01:58:19.156759+0000","last_peered":"2024-04-24T01:58:19.156759+0000","last_clean":"2024-04-24T01:58:19.156759+0000","last_became_active":"2024-04-24T01:55:30.401366+0000","last_became_peered":"2024-04-24T01:55:30.401366+0000","last_unstale":"2024-04-24T01:58:19.156759+0000","last_undegraded":"2024-04-24T01:58:19.156759+0000","last_fullsized":"2024-04-24T01:58:19.156759+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":118,"log_dups_size":0,"ondisk_log_size":118,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":8,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655539,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":80,"num_read_kb":65582,"num_write":37,"num_write_kb":9692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":449,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-04-24T01:58:18.159503+0000","last_change":"2024-04-24T01:55:18.230893+0000","last_active":"2024-04-24T01:58:18.159503+0000","last_peered":"2024-04-24T01:58:18.159503+0000","last_clean":"2024-04-24T01:58:18.159503+0000","last_became_active":"2024-04-24T01:55:18.225188+0000","last_became_peered":"2024-04-24T01:55:18.225188+0000","last_unstale":"2024-04-24T01:58:18.159503+0000","last_undegraded":"2024-04-24T01:58:18.159503+0000","last_fullsized":"2024-04-24T01:58:18.159503+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"182'188","reported_seq":674,"reported_epoch":182,"state":"active+clean+snaptrim","last_fresh":"2024-04-24T01:58:18.219700+0000","last_change":"2024-04-24T01:58:18.196869+0000","last_active":"2024-04-24T01:58:18.219700+0000","last_peered":"2024-04-24T01:58:18.219700+0000","last_clean":"2024-04-24T01:58:18.219700+0000","last_became_active":"2024-04-24T01:55:30.343318+0000","last_became_peered":"2024-04-24T01:55:30.343318+0000","last_unstale":"2024-04-24T01:58:18.219700+0000","last_undegraded":"2024-04-24T01:58:18.219700+0000","last_fullsized":"2024-04-24T01:58:18.219700+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":188,"log_dups_size":0,"ondisk_log_size":188,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":15,"snaptrim_duration":0,"stat_sum":{"num_bytes":6353112,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":90,"num_read_kb":79021,"num_write":78,"num_write_kb":22281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"178'237","reported_seq":733,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-04-24T01:58:18.167258+0000","last_change":"2024-04-24T01:58:18.167258+0000","last_active":"2024-04-24T01:58:18.167258+0000","last_peered":"2024-04-24T01:58:18.167258+0000","last_clean":"2024-04-24T01:58:18.167258+0000","last_became_active":"2024-04-24T01:55:30.345308+0000","last_became_peered":"2024-04-24T01:55:30.345308+0000","last_unstale":"2024-04-24T01:58:18.167258+0000","last_undegraded":"2024-04-24T01:58:18.167258+0000","last_fullsized":"2024-04-24T01:58:18.167258+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":237,"log_dups_size":0,"ondisk_log_size":237,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":13,"snaptrim_duration":0,"stat_sum":{"num_bytes":5764801,"num_objects":8,"num_object_clones":4,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":136,"num_read_kb":94830,"num_write":97,"num_write_kb":23947,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":437,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-04-24T01:58:18.221320+0000","last_change":"2024-04-24T01:55:18.224238+0000","last_active":"2024-04-24T01:58:18.221320+0000","last_peered":"2024-04-24T01:58:18.221320+0000","last_clean":"2024-04-24T01:58:18.221320+0000","last_became_active":"2024-04-24T01:55:18.219131+0000","last_became_peered":"2024-04-24T01:55:18.219131+0000","last_unstale":"2024-04-24T01:58:18.221320+0000","last_undegraded":"2024-04-24T01:58:18.221320+0000","last_fullsized":"2024-04-24T01:58:18.221320+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"174'54","reported_seq":562,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-04-24T01:58:18.222100+0000","last_change":"2024-04-24T01:58:18.292846+0000","last_active":"2024-04-24T01:58:18.222100+0000","last_peered":"2024-04-24T01:58:18.222100+0000","last_clean":"2024-04-24T01:58:18.222100+0000","last_became_active":"2024-04-24T01:55:30.319958+0000","last_became_peered":"2024-04-24T01:55:30.319958+0000","last_unstale":"2024-04-24T01:58:18.222100+0000","last_undegraded":"2024-04-24T01:58:18.222100+0000","last_fullsized":"2024-04-24T01:58:18.222100+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":5,"snaptrim_duration":0,"stat_sum":{"num_bytes":2823609,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":26,"num_read_kb":19608,"num_write":37,"num_write_kb":10363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":452,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-04-24T01:58:19.156997+0000","last_change":"2024-04-24T01:55:18.235763+0000","last_active":"2024-04-24T01:58:19.156997+0000","last_peered":"2024-04-24T01:58:19.156997+0000","last_clean":"2024-04-24T01:58:19.156997+0000","last_became_active":"2024-04-24T01:55:18.229175+0000","last_became_peered":"2024-04-24T01:55:18.229175+0000","last_unstale":"2024-04-24T01:58:19.156997+0000","last_undegraded":"2024-04-24T01:58:19.156997+0000","last_fullsized":"2024-04-24T01:58:19.156997+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"182'81","reported_seq":600,"reported_epoch":183,"state":"active+clean+snaptrim","last_fresh":"2024-04-24T01:58:19.158418+0000","last_change":"2024-04-24T01:58:19.158418+0000","last_active":"2024-04-24T01:58:19.158418+0000","last_peered":"2024-04-24T01:58:19.158418+0000","last_clean":"2024-04-24T01:58:19.158418+0000","last_became_active":"2024-04-24T01:55:30.372690+0000","last_became_peered":"2024-04-24T01:55:30.372690+0000","last_unstale":"2024-04-24T01:58:19.158418+0000","last_undegraded":"2024-04-24T01:58:19.158418+0000","last_fullsized":"2024-04-24T01:58:19.158418+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":5,"snaptrim_duration":0,"stat_sum":{"num_bytes":6621991,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":45,"num_read_kb":30459,"num_write":38,"num_write_kb":11003,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":437,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-04-24T01:58:18.212660+0000","last_change":"2024-04-24T01:55:18.278351+0000","last_active":"2024-04-24T01:58:18.212660+0000","last_peered":"2024-04-24T01:58:18.212660+0000","last_clean":"2024-04-24T01:58:18.212660+0000","last_became_active":"2024-04-24T01:55:18.273668+0000","last_became_peered":"2024-04-24T01:55:18.273668+0000","last_unstale":"2024-04-24T01:58:18.212660+0000","last_undegraded":"2024-04-24T01:58:18.212660+0000","last_fullsized":"2024-04-24T01:58:18.212660+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"182'190","reported_seq":696,"reported_epoch":182,"state":"active+clean+snaptrim","last_fresh":"2024-04-24T01:58:18.249612+0000","last_change":"2024-04-24T01:58:18.202190+0000","last_active":"2024-04-24T01:58:18.249612+0000","last_peered":"2024-04-24T01:58:18.249612+0000","last_clean":"2024-04-24T01:58:18.249612+0000","last_became_active":"2024-04-24T01:55:30.363123+0000","last_became_peered":"2024-04-24T01:55:30.363123+0000","last_unstale":"2024-04-24T01:58:18.249612+0000","last_undegraded":"2024-04-24T01:58:18.249612+0000","last_fullsized":"2024-04-24T01:58:18.249612+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":190,"log_dups_size":0,"ondisk_log_size":190,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":18,"snaptrim_duration":0,"stat_sum":{"num_bytes":5949711,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":118,"num_read_kb":95786,"num_write":98,"num_write_kb":26149,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":443,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-04-24T01:58:18.189730+0000","last_change":"2024-04-24T01:55:18.241117+0000","last_active":"2024-04-24T01:58:18.189730+0000","last_peered":"2024-04-24T01:58:18.189730+0000","last_clean":"2024-04-24T01:58:18.189730+0000","last_became_active":"2024-04-24T01:55:18.234313+0000","last_became_peered":"2024-04-24T01:55:18.234313+0000","last_unstale":"2024-04-24T01:58:18.189730+0000","last_undegraded":"2024-04-24T01:58:18.189730+0000","last_fullsized":"2024-04-24T01:58:18.189730+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":95128346,"num_objects":55,"num_object_clones":19,"num_object_copies":165,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":1354,"num_read_kb":955103,"num_write":1175,"num_write_kb":321600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":242225152,"data_stored":251118505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2602,"ondisk_log_size":2602,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":1}],"osd_stats":[{"osd":3,"up_from":10,"seq":42949673144,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":122176,"kb_used_data":85396,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94249664,"statfs":{"total":96636764160,"available":96511655936,"internally_reserved":0,"allocated":87445504,"data_stored":90906707,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673144,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":103744,"kb_used_data":69640,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94268096,"statfs":{"total":96636764160,"available":96530530304,"internally_reserved":0,"allocated":71311360,"data_stored":76072089,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705849,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":105616,"kb_used_data":73520,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94266224,"statfs":{"total":96636764160,"available":96528613376,"internally_reserved":0,"allocated":75284480,"data_stored":76917194,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705848,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123140,"kb_used_data":92156,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94248700,"statfs":{"total":96636764160,"available":96510668800,"internally_reserved":0,"allocated":94367744,"data_stored":97714725,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":47063040,"data_stored":47442298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":92889088,"data_stored":97076333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":44331008,"data_stored":46734626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":57942016,"data_stored":59865248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T01:58:21.144 INFO:tasks.rados.rados.0.smithi079.stdout:721: done (2 left) 2024-04-24T01:58:21.144 INFO:tasks.rados.rados.0.smithi079.stdout:722: done (1 left) 2024-04-24T01:58:21.144 INFO:tasks.rados.rados.0.smithi079.stdout:723: done (0 left) 2024-04-24T01:58:21.144 INFO:tasks.rados.rados.0.smithi079.stdout:724: snap_create 2024-04-24T01:58:22.029 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T01:58:22.145 INFO:tasks.rados.rados.0.smithi079.stdout:724: done (0 left) 2024-04-24T01:58:22.146 INFO:tasks.rados.rados.0.smithi079.stdout:725: delete oid 15 current snap is 85 2024-04-24T01:58:22.149 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:58:22.146+0000 7f8cd0ead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:22.154 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:58:22.153+0000 7f8cd0ead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:22.227 INFO:tasks.rados.rados.0.smithi079.stdout:725: done (0 left) 2024-04-24T01:58:22.227 INFO:tasks.rados.rados.0.smithi079.stdout:726: write oid 18 current snap is 85 2024-04-24T01:58:22.228 INFO:tasks.rados.rados.0.smithi079.stdout:726: seq_num 351 ranges {699257=564637,1873062=640438,3305941=593316} 2024-04-24T01:58:22.234 INFO:tasks.rados.rados.0.smithi079.stdout:726: writing smithi07929762-18 from 699257 to 1263894 tid 1 2024-04-24T01:58:22.242 INFO:tasks.rados.rados.0.smithi079.stdout:726: writing smithi07929762-18 from 1873062 to 2513500 tid 2 2024-04-24T01:58:22.250 INFO:tasks.rados.rados.0.smithi079.stdout:726: writing smithi07929762-18 from 3305941 to 3899257 tid 3 2024-04-24T01:58:22.250 INFO:tasks.rados.rados.0.smithi079.stdout:727: snap_create 2024-04-24T01:58:22.322 INFO:tasks.rados.rados.0.smithi079.stdout:726: finishing write tid 1 to smithi07929762-18 2024-04-24T01:58:22.331 INFO:tasks.rados.rados.0.smithi079.stdout:726: finishing write tid 2 to smithi07929762-18 2024-04-24T01:58:22.338 INFO:tasks.rados.rados.0.smithi079.stdout:726: finishing write tid 3 to smithi07929762-18 2024-04-24T01:58:22.343 INFO:tasks.rados.rados.0.smithi079.stdout:726: finishing write tid 4 to smithi07929762-18 2024-04-24T01:58:22.344 INFO:tasks.rados.rados.0.smithi079.stdout:726: finishing write tid 5 to smithi07929762-18 2024-04-24T01:58:22.344 INFO:tasks.rados.rados.0.smithi079.stdout:726: oid 18 updating version 0 to 404 2024-04-24T01:58:22.345 INFO:tasks.rados.rados.0.smithi079.stdout:726: oid 18 version 404 is already newer than 404 2024-04-24T01:58:22.345 INFO:tasks.rados.rados.0.smithi079.stdout:726: oid 18 version 404 is already newer than 404 2024-04-24T01:58:22.345 INFO:tasks.rados.rados.0.smithi079.stdout:726: oid 18 version 404 is already newer than 404 2024-04-24T01:58:22.345 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 18 v 404 (ObjNum 351 snap 85 seq_num 351) dirty exists 2024-04-24T01:58:22.345 INFO:tasks.rados.rados.0.smithi079.stdout:726: left oid 18 (ObjNum 351 snap 85 seq_num 351) 2024-04-24T01:58:23.151 INFO:tasks.rados.rados.0.smithi079.stdout:726: done (1 left) 2024-04-24T01:58:23.152 INFO:tasks.rados.rados.0.smithi079.stdout:727: done (0 left) 2024-04-24T01:58:23.152 INFO:tasks.rados.rados.0.smithi079.stdout:728: write (excl) oid 19 current snap is 86 2024-04-24T01:58:23.153 INFO:tasks.rados.rados.0.smithi079.stdout:728: seq_num 352 ranges {716064=639886,1815191=500449,2817011=526925} 2024-04-24T01:58:23.158 INFO:tasks.rados.rados.0.smithi079.stdout:728: writing smithi07929762-19 from 716064 to 1355950 tid 1 2024-04-24T01:58:23.163 INFO:tasks.rados.rados.0.smithi079.stdout:728: writing smithi07929762-19 from 1815191 to 2315640 tid 2 2024-04-24T01:58:23.169 INFO:tasks.rados.rados.0.smithi079.stdout:728: writing smithi07929762-19 from 2817011 to 3343936 tid 3 2024-04-24T01:58:23.169 INFO:tasks.rados.rados.0.smithi079.stdout:729: write oid 9 current snap is 86 2024-04-24T01:58:23.170 INFO:tasks.rados.rados.0.smithi079.stdout:729: seq_num 353 ranges {732871=715135,1873673=644107,3244434=688437} 2024-04-24T01:58:23.177 INFO:tasks.rados.rados.0.smithi079.stdout:729: writing smithi07929762-9 from 732871 to 1448006 tid 1 2024-04-24T01:58:23.185 INFO:tasks.rados.rados.0.smithi079.stdout:729: writing smithi07929762-9 from 1873673 to 2517780 tid 2 2024-04-24T01:58:23.196 INFO:tasks.rados.rados.0.smithi079.stdout:729: writing smithi07929762-9 from 3244434 to 3932871 tid 3 2024-04-24T01:58:23.196 INFO:tasks.rados.rados.0.smithi079.stdout:730: delete oid 5 current snap is 86 2024-04-24T01:58:23.230 INFO:tasks.rados.rados.0.smithi079.stdout:728: finishing write tid 1 to smithi07929762-19 2024-04-24T01:58:23.254 INFO:tasks.rados.rados.0.smithi079.stdout:728: finishing write tid 2 to smithi07929762-19 2024-04-24T01:58:23.285 INFO:tasks.rados.rados.0.smithi079.stdout:728: finishing write tid 3 to smithi07929762-19 2024-04-24T01:58:23.296 INFO:tasks.rados.rados.0.smithi079.stdout:728: finishing write tid 4 to smithi07929762-19 2024-04-24T01:58:23.300 INFO:tasks.rados.rados.0.smithi079.stdout:728: finishing write tid 5 to smithi07929762-19 2024-04-24T01:58:23.300 INFO:tasks.rados.rados.0.smithi079.stdout:728: oid 19 updating version 0 to 233 2024-04-24T01:58:23.301 INFO:tasks.rados.rados.0.smithi079.stdout:728: oid 19 version 233 is already newer than 233 2024-04-24T01:58:23.301 INFO:tasks.rados.rados.0.smithi079.stdout:728: oid 19 version 233 is already newer than 230 2024-04-24T01:58:23.301 INFO:tasks.rados.rados.0.smithi079.stdout:728: oid 19 version 233 is already newer than 233 2024-04-24T01:58:23.301 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 19 v 233 (ObjNum 352 snap 86 seq_num 352) dirty exists 2024-04-24T01:58:23.301 INFO:tasks.rados.rados.0.smithi079.stdout:728: left oid 19 (ObjNum 352 snap 86 seq_num 352) 2024-04-24T01:58:23.302 INFO:tasks.rados.rados.0.smithi079.stdout:728: done (2 left) 2024-04-24T01:58:23.302 INFO:tasks.rados.rados.0.smithi079.stdout:730: done (1 left) 2024-04-24T01:58:23.302 INFO:tasks.rados.rados.0.smithi079.stdout:731: snap_remove snap 82 2024-04-24T01:58:24.155 INFO:tasks.rados.rados.0.smithi079.stdout:731: done (1 left) 2024-04-24T01:58:24.155 INFO:tasks.rados.rados.0.smithi079.stdout:732: read oid 10 snap -1 2024-04-24T01:58:24.156 INFO:tasks.rados.rados.0.smithi079.stdout:732: expect (ObjNum 338 snap 81 seq_num 338) 2024-04-24T01:58:24.156 INFO:tasks.rados.rados.0.smithi079.stdout:733: snap_remove snap 85 2024-04-24T01:58:24.301 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T01:58:24.406 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:58:24.405+0000 7f7ae502d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:24.410 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:58:24.410+0000 7f7ae502d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:25.162 INFO:tasks.rados.rados.0.smithi079.stdout:733: done (2 left) 2024-04-24T01:58:25.162 INFO:tasks.rados.rados.0.smithi079.stdout:734: delete oid 24 current snap is 86 2024-04-24T01:58:25.162 INFO:tasks.rados.rados.0.smithi079.stdout:729: finishing write tid 1 to smithi07929762-9 2024-04-24T01:58:25.162 INFO:tasks.rados.rados.0.smithi079.stdout:729: finishing write tid 2 to smithi07929762-9 2024-04-24T01:58:25.162 INFO:tasks.rados.rados.0.smithi079.stdout:729: finishing write tid 3 to smithi07929762-9 2024-04-24T01:58:25.163 INFO:tasks.rados.rados.0.smithi079.stdout:729: finishing write tid 4 to smithi07929762-9 2024-04-24T01:58:25.163 INFO:tasks.rados.rados.0.smithi079.stdout:729: finishing write tid 5 to smithi07929762-9 2024-04-24T01:58:25.163 INFO:tasks.rados.rados.0.smithi079.stdout:729: oid 9 updating version 0 to 255 2024-04-24T01:58:25.163 INFO:tasks.rados.rados.0.smithi079.stdout:729: oid 9 version 255 is already newer than 255 2024-04-24T01:58:25.163 INFO:tasks.rados.rados.0.smithi079.stdout:729: oid 9 version 255 is already newer than 255 2024-04-24T01:58:25.163 INFO:tasks.rados.rados.0.smithi079.stdout:729: oid 9 version 255 is already newer than 255 2024-04-24T01:58:25.163 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 9 v 255 (ObjNum 353 snap 86 seq_num 353) dirty exists 2024-04-24T01:58:25.164 INFO:tasks.rados.rados.0.smithi079.stdout:729: left oid 9 (ObjNum 353 snap 86 seq_num 353) 2024-04-24T01:58:25.164 INFO:tasks.rados.rados.0.smithi079.stdout:732: expect (ObjNum 338 snap 81 seq_num 338) 2024-04-24T01:58:25.367 INFO:tasks.rados.rados.0.smithi079.stdout:729: done (2 left) 2024-04-24T01:58:25.368 INFO:tasks.rados.rados.0.smithi079.stdout:732: done (1 left) 2024-04-24T01:58:25.368 INFO:tasks.rados.rados.0.smithi079.stdout:734: done (0 left) 2024-04-24T01:58:25.368 INFO:tasks.rados.rados.0.smithi079.stdout:735: snap_create 2024-04-24T01:58:25.646 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-04-24T01:58:25.646 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-04-24T01:58:25.646 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T01:58:25.754 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:58:25.751+0000 7f2a90d5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:25.758 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:58:25.758+0000 7f2a90d5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:25.943 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:58:25.944 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-24T01:58:25.956 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":276,"stamp":"2024-04-24T01:58:25.157707+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":97886982,"num_objects":36,"num_object_clones":3,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":1488,"num_read_kb":974656,"num_write":1356,"num_write_kb":337578,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2780,"ondisk_log_size":2780,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":312216,"kb_used_data":187652,"kb_used_omap":0,"kb_used_meta":123904,"kb_avail":377175144,"statfs":{"total":386547056640,"available":386227347456,"internally_reserved":0,"allocated":192155648,"data_stored":193893044,"data_compressed":89889,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":126877696},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2168268,"num_objects":-21,"num_object_clones":-16,"num_object_copies":-63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":-5,"num_read":31,"num_read_kb":19462,"num_write":44,"num_write_kb":14084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.015154"},"pg_stats":[{"pgid":"3.f","version":"182'127","reported_seq":666,"reported_epoch":188,"state":"active+clean+snaptrim","last_fresh":"2024-04-24T01:58:24.169455+0000","last_change":"2024-04-24T01:58:24.169455+0000","last_active":"2024-04-24T01:58:24.169455+0000","last_peered":"2024-04-24T01:58:24.169455+0000","last_clean":"2024-04-24T01:58:24.169455+0000","last_became_active":"2024-04-24T01:55:30.386726+0000","last_became_peered":"2024-04-24T01:55:30.386726+0000","last_unstale":"2024-04-24T01:58:24.169455+0000","last_undegraded":"2024-04-24T01:58:24.169455+0000","last_fullsized":"2024-04-24T01:58:24.169455+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":10,"snaptrim_duration":0,"stat_sum":{"num_bytes":6033779,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":92,"num_read_kb":82875,"num_write":53,"num_write_kb":15604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"184'224","reported_seq":735,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-04-24T01:58:24.144430+0000","last_change":"2024-04-24T01:58:20.174005+0000","last_active":"2024-04-24T01:58:24.144430+0000","last_peered":"2024-04-24T01:58:24.144430+0000","last_clean":"2024-04-24T01:58:24.144430+0000","last_became_active":"2024-04-24T01:55:30.321866+0000","last_became_peered":"2024-04-24T01:55:30.321866+0000","last_unstale":"2024-04-24T01:58:24.144430+0000","last_undegraded":"2024-04-24T01:58:24.144430+0000","last_fullsized":"2024-04-24T01:58:24.144430+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":224,"log_dups_size":0,"ondisk_log_size":224,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":16,"snaptrim_duration":0,"stat_sum":{"num_bytes":10353211,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":126,"num_read_kb":82519,"num_write":101,"num_write_kb":27747,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"182'185","reported_seq":681,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-04-24T01:58:22.172054+0000","last_change":"2024-04-24T01:58:19.169682+0000","last_active":"2024-04-24T01:58:22.172054+0000","last_peered":"2024-04-24T01:58:22.172054+0000","last_clean":"2024-04-24T01:58:22.172054+0000","last_became_active":"2024-04-24T01:55:30.320186+0000","last_became_peered":"2024-04-24T01:55:30.320186+0000","last_unstale":"2024-04-24T01:58:22.172054+0000","last_undegraded":"2024-04-24T01:58:22.172054+0000","last_fullsized":"2024-04-24T01:58:22.172054+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":185,"log_dups_size":0,"ondisk_log_size":185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":17,"snaptrim_duration":0,"stat_sum":{"num_bytes":6353112,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":64,"num_read_kb":35668,"num_write":95,"num_write_kb":25777,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"187'195","reported_seq":715,"reported_epoch":188,"state":"active+clean+snaptrim","last_fresh":"2024-04-24T01:58:24.168437+0000","last_change":"2024-04-24T01:58:24.168437+0000","last_active":"2024-04-24T01:58:24.168437+0000","last_peered":"2024-04-24T01:58:24.168437+0000","last_clean":"2024-04-24T01:58:24.168437+0000","last_became_active":"2024-04-24T01:55:30.288081+0000","last_became_peered":"2024-04-24T01:55:30.288081+0000","last_unstale":"2024-04-24T01:58:24.168437+0000","last_undegraded":"2024-04-24T01:58:24.168437+0000","last_fullsized":"2024-04-24T01:58:24.168437+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":195,"log_dups_size":0,"ondisk_log_size":195,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":18,"snaptrim_duration":0,"stat_sum":{"num_bytes":7109427,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":107,"num_read_kb":77554,"num_write":81,"num_write_kb":23066,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"182'61","reported_seq":596,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-04-24T01:58:23.177253+0000","last_change":"2024-04-24T01:58:19.164226+0000","last_active":"2024-04-24T01:58:23.177253+0000","last_peered":"2024-04-24T01:58:23.177253+0000","last_clean":"2024-04-24T01:58:23.177253+0000","last_became_active":"2024-04-24T01:55:30.333815+0000","last_became_peered":"2024-04-24T01:55:30.333815+0000","last_unstale":"2024-04-24T01:58:23.177253+0000","last_undegraded":"2024-04-24T01:58:23.177253+0000","last_fullsized":"2024-04-24T01:58:23.177253+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":5,"snaptrim_duration":0,"stat_sum":{"num_bytes":3798382,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":32,"num_read_kb":17556,"num_write":36,"num_write_kb":10196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"186'404","reported_seq":880,"reported_epoch":188,"state":"active+clean+snaptrim","last_fresh":"2024-04-24T01:58:24.166258+0000","last_change":"2024-04-24T01:58:24.166258+0000","last_active":"2024-04-24T01:58:24.166258+0000","last_peered":"2024-04-24T01:58:24.166258+0000","last_clean":"2024-04-24T01:58:24.166258+0000","last_became_active":"2024-04-24T01:55:30.315075+0000","last_became_peered":"2024-04-24T01:55:30.315075+0000","last_unstale":"2024-04-24T01:58:24.166258+0000","last_undegraded":"2024-04-24T01:58:24.166258+0000","last_fullsized":"2024-04-24T01:58:24.166258+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":404,"log_dups_size":0,"ondisk_log_size":404,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":36,"snaptrim_duration":0,"stat_sum":{"num_bytes":7479181,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":170,"num_read_kb":102270,"num_write":167,"num_write_kb":44904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"183'66","reported_seq":600,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-24T01:58:23.236078+0000","last_change":"2024-04-24T01:58:19.262658+0000","last_active":"2024-04-24T01:58:23.236078+0000","last_peered":"2024-04-24T01:58:23.236078+0000","last_clean":"2024-04-24T01:58:23.236078+0000","last_became_active":"2024-04-24T01:55:30.350300+0000","last_became_peered":"2024-04-24T01:55:30.350300+0000","last_unstale":"2024-04-24T01:58:23.236078+0000","last_undegraded":"2024-04-24T01:58:23.236078+0000","last_fullsized":"2024-04-24T01:58:23.236078+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0,"stat_sum":{"num_bytes":3596731,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":33,"num_read_kb":16964,"num_write":41,"num_write_kb":10355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"163'8","reported_seq":529,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-04-24T01:58:22.199549+0000","last_change":"2024-04-24T01:58:19.202286+0000","last_active":"2024-04-24T01:58:22.199549+0000","last_peered":"2024-04-24T01:58:22.199549+0000","last_clean":"2024-04-24T01:58:22.199549+0000","last_became_active":"2024-04-24T01:55:30.376813+0000","last_became_peered":"2024-04-24T01:55:30.376813+0000","last_unstale":"2024-04-24T01:58:22.199549+0000","last_undegraded":"2024-04-24T01:58:22.199549+0000","last_fullsized":"2024-04-24T01:58:22.199549+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'97","reported_seq":597,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-04-24T01:58:23.172569+0000","last_change":"2024-04-24T01:55:16.196470+0000","last_active":"2024-04-24T01:58:23.172569+0000","last_peered":"2024-04-24T01:58:23.172569+0000","last_clean":"2024-04-24T01:58:23.172569+0000","last_became_active":"2024-04-24T01:55:16.190464+0000","last_became_peered":"2024-04-24T01:55:16.190464+0000","last_unstale":"2024-04-24T01:58:23.172569+0000","last_undegraded":"2024-04-24T01:58:23.172569+0000","last_fullsized":"2024-04-24T01:58:23.172569+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:15.127115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:15.127115+0000","last_clean_scrub_stamp":"2024-04-24T01:55:15.127115+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":451,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-04-24T01:58:22.196208+0000","last_change":"2024-04-24T01:55:18.236148+0000","last_active":"2024-04-24T01:58:22.196208+0000","last_peered":"2024-04-24T01:58:22.196208+0000","last_clean":"2024-04-24T01:58:22.196208+0000","last_became_active":"2024-04-24T01:55:18.230154+0000","last_became_peered":"2024-04-24T01:55:18.230154+0000","last_unstale":"2024-04-24T01:58:22.196208+0000","last_undegraded":"2024-04-24T01:58:22.196208+0000","last_fullsized":"2024-04-24T01:58:22.196208+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"183'289","reported_seq":801,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-04-24T01:58:22.203507+0000","last_change":"2024-04-24T01:58:19.266453+0000","last_active":"2024-04-24T01:58:22.203507+0000","last_peered":"2024-04-24T01:58:22.203507+0000","last_clean":"2024-04-24T01:58:22.203507+0000","last_became_active":"2024-04-24T01:55:30.316222+0000","last_became_peered":"2024-04-24T01:55:30.316222+0000","last_unstale":"2024-04-24T01:58:22.203507+0000","last_undegraded":"2024-04-24T01:58:22.203507+0000","last_fullsized":"2024-04-24T01:58:22.203507+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":289,"log_dups_size":0,"ondisk_log_size":289,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":23,"snaptrim_duration":0,"stat_sum":{"num_bytes":11311210,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":157,"num_read_kb":107365,"num_write":142,"num_write_kb":40402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":462,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-24T01:58:23.232171+0000","last_change":"2024-04-24T01:55:18.265905+0000","last_active":"2024-04-24T01:58:23.232171+0000","last_peered":"2024-04-24T01:58:23.232171+0000","last_clean":"2024-04-24T01:58:23.232171+0000","last_became_active":"2024-04-24T01:55:18.257020+0000","last_became_peered":"2024-04-24T01:55:18.257020+0000","last_unstale":"2024-04-24T01:58:23.232171+0000","last_undegraded":"2024-04-24T01:58:23.232171+0000","last_fullsized":"2024-04-24T01:58:23.232171+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"187'233","reported_seq":738,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-04-24T01:58:23.234248+0000","last_change":"2024-04-24T01:58:19.168222+0000","last_active":"2024-04-24T01:58:23.234248+0000","last_peered":"2024-04-24T01:58:23.234248+0000","last_clean":"2024-04-24T01:58:23.234248+0000","last_became_active":"2024-04-24T01:55:30.344129+0000","last_became_peered":"2024-04-24T01:55:30.344129+0000","last_unstale":"2024-04-24T01:58:23.234248+0000","last_undegraded":"2024-04-24T01:58:23.234248+0000","last_fullsized":"2024-04-24T01:58:23.234248+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":233,"log_dups_size":0,"ondisk_log_size":233,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":15,"snaptrim_duration":0,"stat_sum":{"num_bytes":10722965,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":98,"num_read_kb":60233,"num_write":101,"num_write_kb":29665,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":451,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-04-24T01:58:22.175425+0000","last_change":"2024-04-24T01:55:18.215251+0000","last_active":"2024-04-24T01:58:22.175425+0000","last_peered":"2024-04-24T01:58:22.175425+0000","last_clean":"2024-04-24T01:58:22.175425+0000","last_became_active":"2024-04-24T01:55:18.206605+0000","last_became_peered":"2024-04-24T01:55:18.206605+0000","last_unstale":"2024-04-24T01:58:22.175425+0000","last_undegraded":"2024-04-24T01:58:22.175425+0000","last_fullsized":"2024-04-24T01:58:22.175425+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"182'118","reported_seq":649,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-04-24T01:58:23.178166+0000","last_change":"2024-04-24T01:58:19.166791+0000","last_active":"2024-04-24T01:58:23.178166+0000","last_peered":"2024-04-24T01:58:23.178166+0000","last_clean":"2024-04-24T01:58:23.178166+0000","last_became_active":"2024-04-24T01:55:30.401366+0000","last_became_peered":"2024-04-24T01:55:30.401366+0000","last_unstale":"2024-04-24T01:58:23.178166+0000","last_undegraded":"2024-04-24T01:58:23.178166+0000","last_fullsized":"2024-04-24T01:58:23.178166+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":118,"log_dups_size":0,"ondisk_log_size":118,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":8,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655539,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":80,"num_read_kb":65582,"num_write":37,"num_write_kb":9692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":462,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-24T01:58:23.233770+0000","last_change":"2024-04-24T01:55:18.230893+0000","last_active":"2024-04-24T01:58:23.233770+0000","last_peered":"2024-04-24T01:58:23.233770+0000","last_clean":"2024-04-24T01:58:23.233770+0000","last_became_active":"2024-04-24T01:55:18.225188+0000","last_became_peered":"2024-04-24T01:55:18.225188+0000","last_unstale":"2024-04-24T01:58:23.233770+0000","last_undegraded":"2024-04-24T01:58:23.233770+0000","last_fullsized":"2024-04-24T01:58:23.233770+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"183'194","reported_seq":695,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-04-24T01:58:24.142118+0000","last_change":"2024-04-24T01:58:19.197655+0000","last_active":"2024-04-24T01:58:24.142118+0000","last_peered":"2024-04-24T01:58:24.142118+0000","last_clean":"2024-04-24T01:58:24.142118+0000","last_became_active":"2024-04-24T01:55:30.343318+0000","last_became_peered":"2024-04-24T01:55:30.343318+0000","last_unstale":"2024-04-24T01:58:24.142118+0000","last_undegraded":"2024-04-24T01:58:24.142118+0000","last_fullsized":"2024-04-24T01:58:24.142118+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":194,"log_dups_size":0,"ondisk_log_size":194,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":15,"snaptrim_duration":0,"stat_sum":{"num_bytes":6974971,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":91,"num_read_kb":79022,"num_write":83,"num_write_kb":24096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"183'250","reported_seq":755,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-04-24T01:58:22.201685+0000","last_change":"2024-04-24T01:58:19.201285+0000","last_active":"2024-04-24T01:58:22.201685+0000","last_peered":"2024-04-24T01:58:22.201685+0000","last_clean":"2024-04-24T01:58:22.201685+0000","last_became_active":"2024-04-24T01:55:30.345308+0000","last_became_peered":"2024-04-24T01:55:30.345308+0000","last_unstale":"2024-04-24T01:58:22.201685+0000","last_undegraded":"2024-04-24T01:58:22.201685+0000","last_fullsized":"2024-04-24T01:58:22.201685+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":250,"log_dups_size":0,"ondisk_log_size":250,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":19,"snaptrim_duration":0,"stat_sum":{"num_bytes":8823774,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":138,"num_read_kb":94831,"num_write":103,"num_write_kb":25817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":450,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-04-24T01:58:24.154827+0000","last_change":"2024-04-24T01:55:18.224238+0000","last_active":"2024-04-24T01:58:24.154827+0000","last_peered":"2024-04-24T01:58:24.154827+0000","last_clean":"2024-04-24T01:58:24.154827+0000","last_became_active":"2024-04-24T01:55:18.219131+0000","last_became_peered":"2024-04-24T01:55:18.219131+0000","last_unstale":"2024-04-24T01:58:24.154827+0000","last_undegraded":"2024-04-24T01:58:24.154827+0000","last_fullsized":"2024-04-24T01:58:24.154827+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"174'54","reported_seq":576,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-04-24T01:58:24.157854+0000","last_change":"2024-04-24T01:58:19.197822+0000","last_active":"2024-04-24T01:58:24.157854+0000","last_peered":"2024-04-24T01:58:24.157854+0000","last_clean":"2024-04-24T01:58:24.157854+0000","last_became_active":"2024-04-24T01:55:30.319958+0000","last_became_peered":"2024-04-24T01:55:30.319958+0000","last_unstale":"2024-04-24T01:58:24.157854+0000","last_undegraded":"2024-04-24T01:58:24.157854+0000","last_fullsized":"2024-04-24T01:58:24.157854+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":5,"snaptrim_duration":0,"stat_sum":{"num_bytes":2823609,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":26,"num_read_kb":19608,"num_write":37,"num_write_kb":10363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":464,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-04-24T01:58:23.179000+0000","last_change":"2024-04-24T01:55:18.235763+0000","last_active":"2024-04-24T01:58:23.179000+0000","last_peered":"2024-04-24T01:58:23.179000+0000","last_clean":"2024-04-24T01:58:23.179000+0000","last_became_active":"2024-04-24T01:55:18.229175+0000","last_became_peered":"2024-04-24T01:55:18.229175+0000","last_unstale":"2024-04-24T01:58:23.179000+0000","last_undegraded":"2024-04-24T01:58:23.179000+0000","last_fullsized":"2024-04-24T01:58:23.179000+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"182'81","reported_seq":613,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-04-24T01:58:23.170746+0000","last_change":"2024-04-24T01:58:19.170387+0000","last_active":"2024-04-24T01:58:23.170746+0000","last_peered":"2024-04-24T01:58:23.170746+0000","last_clean":"2024-04-24T01:58:23.170746+0000","last_became_active":"2024-04-24T01:55:30.372690+0000","last_became_peered":"2024-04-24T01:55:30.372690+0000","last_unstale":"2024-04-24T01:58:23.170746+0000","last_undegraded":"2024-04-24T01:58:23.170746+0000","last_fullsized":"2024-04-24T01:58:23.170746+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":5,"snaptrim_duration":0,"stat_sum":{"num_bytes":6621991,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":45,"num_read_kb":30459,"num_write":38,"num_write_kb":11003,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":450,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-04-24T01:58:24.155180+0000","last_change":"2024-04-24T01:55:18.278351+0000","last_active":"2024-04-24T01:58:24.155180+0000","last_peered":"2024-04-24T01:58:24.155180+0000","last_clean":"2024-04-24T01:58:24.155180+0000","last_became_active":"2024-04-24T01:55:18.273668+0000","last_became_peered":"2024-04-24T01:55:18.273668+0000","last_unstale":"2024-04-24T01:58:24.155180+0000","last_undegraded":"2024-04-24T01:58:24.155180+0000","last_fullsized":"2024-04-24T01:58:24.155180+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"186'194","reported_seq":716,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-04-24T01:58:24.156071+0000","last_change":"2024-04-24T01:58:19.202532+0000","last_active":"2024-04-24T01:58:24.156071+0000","last_peered":"2024-04-24T01:58:24.156071+0000","last_clean":"2024-04-24T01:58:24.156071+0000","last_became_active":"2024-04-24T01:55:30.363123+0000","last_became_peered":"2024-04-24T01:55:30.363123+0000","last_unstale":"2024-04-24T01:58:24.156071+0000","last_undegraded":"2024-04-24T01:58:24.156071+0000","last_fullsized":"2024-04-24T01:58:24.156071+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":194,"log_dups_size":0,"ondisk_log_size":194,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":18,"snaptrim_duration":0,"stat_sum":{"num_bytes":2638732,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":124,"num_read_kb":102058,"num_write":99,"num_write_kb":26149,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":451,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-04-24T01:58:22.202375+0000","last_change":"2024-04-24T01:55:18.241117+0000","last_active":"2024-04-24T01:58:22.202375+0000","last_peered":"2024-04-24T01:58:22.202375+0000","last_clean":"2024-04-24T01:58:22.202375+0000","last_became_active":"2024-04-24T01:55:18.234313+0000","last_became_peered":"2024-04-24T01:55:18.234313+0000","last_unstale":"2024-04-24T01:58:22.202375+0000","last_undegraded":"2024-04-24T01:58:22.202375+0000","last_fullsized":"2024-04-24T01:58:22.202375+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":97296614,"num_objects":34,"num_object_clones":3,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":1385,"num_read_kb":974565,"num_write":1219,"num_write_kb":335684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":188465152,"data_stored":191978466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2683,"ondisk_log_size":2683,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":1}],"osd_stats":[{"osd":3,"up_from":10,"seq":42949673149,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":87168,"kb_used_data":56184,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94284672,"statfs":{"total":96636764160,"available":96547504128,"internally_reserved":0,"allocated":57532416,"data_stored":57191641,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673149,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":71968,"kb_used_data":40984,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94299872,"statfs":{"total":96636764160,"available":96563068928,"internally_reserved":0,"allocated":41967616,"data_stored":43225054,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":77120,"kb_used_data":46136,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94294720,"statfs":{"total":96636764160,"available":96557793280,"internally_reserved":0,"allocated":47243264,"data_stored":47862875,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":75960,"kb_used_data":44348,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94295880,"statfs":{"total":96636764160,"available":96558981120,"internally_reserved":0,"allocated":45412352,"data_stored":45613474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":45645824,"data_stored":46608664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":45121536,"data_stored":46081050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":41070592,"data_stored":42662885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":56627200,"data_stored":56625867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T01:58:26.037 INFO:tasks.rados.rados.0.smithi079.stdout:735: done (0 left) 2024-04-24T01:58:26.038 INFO:tasks.rados.rados.0.smithi079.stdout:736: write oid 12 current snap is 87 2024-04-24T01:58:26.038 INFO:tasks.rados.rados.0.smithi079.stdout:736: seq_num 356 ranges {783292=540882,1932766=507787,3210350=572013} 2024-04-24T01:58:26.044 INFO:tasks.rados.rados.0.smithi079.stdout:736: writing smithi07929762-12 from 783292 to 1324174 tid 1 2024-04-24T01:58:26.051 INFO:tasks.rados.rados.0.smithi079.stdout:736: writing smithi07929762-12 from 1932766 to 2440553 tid 2 2024-04-24T01:58:26.058 INFO:tasks.rados.rados.0.smithi079.stdout:736: writing smithi07929762-12 from 3210350 to 3782363 tid 3 2024-04-24T01:58:26.058 INFO:tasks.rados.rados.0.smithi079.stdout:737: snap_remove snap 84 2024-04-24T01:58:26.502 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T01:58:26.607 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:58:26.605+0000 7f97f602d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:26.651 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:58:26.611+0000 7f97f602d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:27.040 INFO:tasks.rados.rados.0.smithi079.stdout:737: done (1 left) 2024-04-24T01:58:27.040 INFO:tasks.rados.rados.0.smithi079.stdout:738: delete oid 31 current snap is 87 2024-04-24T01:58:27.040 INFO:tasks.rados.rados.0.smithi079.stdout:736: finishing write tid 1 to smithi07929762-12 2024-04-24T01:58:27.040 INFO:tasks.rados.rados.0.smithi079.stdout:736: finishing write tid 2 to smithi07929762-12 2024-04-24T01:58:27.040 INFO:tasks.rados.rados.0.smithi079.stdout:736: finishing write tid 3 to smithi07929762-12 2024-04-24T01:58:27.040 INFO:tasks.rados.rados.0.smithi079.stdout:736: finishing write tid 4 to smithi07929762-12 2024-04-24T01:58:27.040 INFO:tasks.rados.rados.0.smithi079.stdout:736: finishing write tid 5 to smithi07929762-12 2024-04-24T01:58:27.040 INFO:tasks.rados.rados.0.smithi079.stdout:736: oid 12 updating version 0 to 200 2024-04-24T01:58:27.040 INFO:tasks.rados.rados.0.smithi079.stdout:736: oid 12 version 200 is already newer than 200 2024-04-24T01:58:27.041 INFO:tasks.rados.rados.0.smithi079.stdout:736: oid 12 version 200 is already newer than 200 2024-04-24T01:58:27.041 INFO:tasks.rados.rados.0.smithi079.stdout:736: oid 12 version 200 is already newer than 200 2024-04-24T01:58:27.041 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 12 v 200 (ObjNum 356 snap 87 seq_num 356) dirty exists 2024-04-24T01:58:27.041 INFO:tasks.rados.rados.0.smithi079.stdout:736: left oid 12 (ObjNum 356 snap 87 seq_num 356) 2024-04-24T01:58:27.096 INFO:tasks.rados.rados.0.smithi079.stdout:736: done (1 left) 2024-04-24T01:58:27.096 INFO:tasks.rados.rados.0.smithi079.stdout:738: done (0 left) 2024-04-24T01:58:27.096 INFO:tasks.rados.rados.0.smithi079.stdout:739: write (excl) oid 50 current snap is 87 2024-04-24T01:58:27.096 INFO:tasks.rados.rados.0.smithi079.stdout:739: seq_num 358 ranges {416906=407733,1366083=511456} 2024-04-24T01:58:27.101 INFO:tasks.rados.rados.0.smithi079.stdout:739: writing smithi07929762-50 from 416906 to 824639 tid 1 2024-04-24T01:58:27.107 INFO:tasks.rados.rados.0.smithi079.stdout:739: writing smithi07929762-50 from 1366083 to 1877539 tid 2 2024-04-24T01:58:27.107 INFO:tasks.rados.rados.0.smithi079.stdout:740: snap_create 2024-04-24T01:58:27.137 INFO:tasks.rados.rados.0.smithi079.stdout:739: finishing write tid 1 to smithi07929762-50 2024-04-24T01:58:27.155 INFO:tasks.rados.rados.0.smithi079.stdout:739: finishing write tid 2 to smithi07929762-50 2024-04-24T01:58:27.163 INFO:tasks.rados.rados.0.smithi079.stdout:739: finishing write tid 3 to smithi07929762-50 2024-04-24T01:58:27.165 INFO:tasks.rados.rados.0.smithi079.stdout:739: finishing write tid 4 to smithi07929762-50 2024-04-24T01:58:27.165 INFO:tasks.rados.rados.0.smithi079.stdout:739: oid 50 updating version 0 to 203 2024-04-24T01:58:27.165 INFO:tasks.rados.rados.0.smithi079.stdout:739: oid 50 updating version 203 to 205 2024-04-24T01:58:27.165 INFO:tasks.rados.rados.0.smithi079.stdout:739: oid 50 version 205 is already newer than 205 2024-04-24T01:58:27.165 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 50 v 205 (ObjNum 358 snap 87 seq_num 358) dirty exists 2024-04-24T01:58:27.165 INFO:tasks.rados.rados.0.smithi079.stdout:739: left oid 50 (ObjNum 358 snap 87 seq_num 358) 2024-04-24T01:58:28.040 INFO:tasks.rados.rados.0.smithi079.stdout:739: done (1 left) 2024-04-24T01:58:28.040 INFO:tasks.rados.rados.0.smithi079.stdout:740: done (0 left) 2024-04-24T01:58:28.040 INFO:tasks.rados.rados.0.smithi079.stdout:741: read oid 17 snap -1 2024-04-24T01:58:28.040 INFO:tasks.rados.rados.0.smithi079.stdout:741: expect deleted 2024-04-24T01:58:28.040 INFO:tasks.rados.rados.0.smithi079.stdout:742: snap_create 2024-04-24T01:58:28.703 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T01:58:28.821 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:58:28.820+0000 7f3d16ced640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:28.825 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:58:28.825+0000 7f3d16ced640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:29.045 INFO:tasks.rados.rados.0.smithi079.stdout:741: done (1 left) 2024-04-24T01:58:29.045 INFO:tasks.rados.rados.0.smithi079.stdout:742: done (0 left) 2024-04-24T01:58:29.045 INFO:tasks.rados.rados.0.smithi079.stdout:743: read oid 29 snap -1 2024-04-24T01:58:29.045 INFO:tasks.rados.rados.0.smithi079.stdout:743: expect (ObjNum 294 snap 78 seq_num 294) 2024-04-24T01:58:29.045 INFO:tasks.rados.rados.0.smithi079.stdout:744: write oid 35 current snap is 89 2024-04-24T01:58:29.045 INFO:tasks.rados.rados.0.smithi079.stdout:744: seq_num 359 ranges {433713=482982,1424565=609148} 2024-04-24T01:58:29.049 INFO:tasks.rados.rados.0.smithi079.stdout:744: writing smithi07929762-35 from 433713 to 916695 tid 1 2024-04-24T01:58:29.057 INFO:tasks.rados.rados.0.smithi079.stdout:744: writing smithi07929762-35 from 1424565 to 2033713 tid 2 2024-04-24T01:58:29.057 INFO:tasks.rados.rados.0.smithi079.stdout:745: read oid 46 snap -1 2024-04-24T01:58:29.057 INFO:tasks.rados.rados.0.smithi079.stdout:745: expect deleted 2024-04-24T01:58:29.057 INFO:tasks.rados.rados.0.smithi079.stdout:746: read oid 9 snap -1 2024-04-24T01:58:29.058 INFO:tasks.rados.rados.0.smithi079.stdout:746: expect (ObjNum 353 snap 86 seq_num 353) 2024-04-24T01:58:29.058 INFO:tasks.rados.rados.0.smithi079.stdout:747: delete oid 30 current snap is 89 2024-04-24T01:58:29.143 INFO:tasks.rados.rados.0.smithi079.stdout:744: finishing write tid 1 to smithi07929762-35 2024-04-24T01:58:29.155 INFO:tasks.rados.rados.0.smithi079.stdout:746: expect (ObjNum 353 snap 86 seq_num 353) 2024-04-24T01:58:29.371 INFO:tasks.rados.rados.0.smithi079.stdout:744: finishing write tid 2 to smithi07929762-35 2024-04-24T01:58:29.371 INFO:tasks.rados.rados.0.smithi079.stdout:744: finishing write tid 3 to smithi07929762-35 2024-04-24T01:58:29.371 INFO:tasks.rados.rados.0.smithi079.stdout:745: done (4 left) 2024-04-24T01:58:29.371 INFO:tasks.rados.rados.0.smithi079.stdout:746: done (3 left) 2024-04-24T01:58:29.371 INFO:tasks.rados.rados.0.smithi079.stdout:747: done (2 left) 2024-04-24T01:58:29.371 INFO:tasks.rados.rados.0.smithi079.stdout:748: write oid 19 current snap is 89 2024-04-24T01:58:29.371 INFO:tasks.rados.rados.0.smithi079.stdout:743: expect (ObjNum 294 snap 78 seq_num 294) 2024-04-24T01:58:29.524 INFO:tasks.rados.rados.0.smithi079.stdout:744: finishing write tid 4 to smithi07929762-35 2024-04-24T01:58:29.524 INFO:tasks.rados.rados.0.smithi079.stdout:744: oid 35 updating version 0 to 241 2024-04-24T01:58:29.524 INFO:tasks.rados.rados.0.smithi079.stdout:744: oid 35 version 241 is already newer than 241 2024-04-24T01:58:29.524 INFO:tasks.rados.rados.0.smithi079.stdout:744: oid 35 version 241 is already newer than 241 2024-04-24T01:58:29.524 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 35 v 241 (ObjNum 359 snap 89 seq_num 359) dirty exists 2024-04-24T01:58:29.525 INFO:tasks.rados.rados.0.smithi079.stdout:744: left oid 35 (ObjNum 359 snap 89 seq_num 359) 2024-04-24T01:58:29.525 INFO:tasks.rados.rados.0.smithi079.stdout:748: seq_num 361 ranges {467327=633480,1825176=242151} 2024-04-24T01:58:29.530 INFO:tasks.rados.rados.0.smithi079.stdout:748: writing smithi07929762-19 from 467327 to 1100807 tid 1 2024-04-24T01:58:29.535 INFO:tasks.rados.rados.0.smithi079.stdout:748: writing smithi07929762-19 from 1825176 to 2067327 tid 2 2024-04-24T01:58:29.536 INFO:tasks.rados.rados.0.smithi079.stdout:743: done (2 left) 2024-04-24T01:58:29.536 INFO:tasks.rados.rados.0.smithi079.stdout:744: done (1 left) 2024-04-24T01:58:29.536 INFO:tasks.rados.rados.0.smithi079.stdout:749: read oid 5 snap 88 2024-04-24T01:58:29.536 INFO:tasks.rados.rados.0.smithi079.stdout:749: expect deleted 2024-04-24T01:58:29.536 INFO:tasks.rados.rados.0.smithi079.stdout:750: write oid 25 current snap is 89 2024-04-24T01:58:29.536 INFO:tasks.rados.rados.0.smithi079.stdout:750: seq_num 362 ranges {484134=708729,1883658=200476} 2024-04-24T01:58:29.542 INFO:tasks.rados.rados.0.smithi079.stdout:750: writing smithi07929762-25 from 484134 to 1192863 tid 1 2024-04-24T01:58:29.547 INFO:tasks.rados.rados.0.smithi079.stdout:750: writing smithi07929762-25 from 1883658 to 2084134 tid 2 2024-04-24T01:58:29.547 INFO:tasks.rados.rados.0.smithi079.stdout:751: write (excl) oid 44 current snap is 89 2024-04-24T01:58:29.547 INFO:tasks.rados.rados.0.smithi079.stdout:751: seq_num 363 ranges {500941=783978,1942140=158801} 2024-04-24T01:58:29.556 INFO:tasks.rados.rados.0.smithi079.stdout:751: writing smithi07929762-44 from 500941 to 1284919 tid 1 2024-04-24T01:58:29.562 INFO:tasks.rados.rados.0.smithi079.stdout:751: writing smithi07929762-44 from 1942140 to 2100941 tid 2 2024-04-24T01:58:29.562 INFO:tasks.rados.rados.0.smithi079.stdout:752: delete oid 22 current snap is 89 2024-04-24T01:58:29.617 INFO:tasks.rados.rados.0.smithi079.stdout:748: finishing write tid 1 to smithi07929762-19 2024-04-24T01:58:29.630 INFO:tasks.rados.rados.0.smithi079.stdout:748: finishing write tid 2 to smithi07929762-19 2024-04-24T01:58:29.636 INFO:tasks.rados.rados.0.smithi079.stdout:750: finishing write tid 1 to smithi07929762-25 2024-04-24T01:58:29.636 INFO:tasks.rados.rados.0.smithi079.stdout:748: finishing write tid 3 to smithi07929762-19 2024-04-24T01:58:29.648 INFO:tasks.rados.rados.0.smithi079.stdout:748: finishing write tid 4 to smithi07929762-19 2024-04-24T01:58:29.648 INFO:tasks.rados.rados.0.smithi079.stdout:748: oid 19 updating version 0 to 245 2024-04-24T01:58:29.648 INFO:tasks.rados.rados.0.smithi079.stdout:748: oid 19 version 245 is already newer than 245 2024-04-24T01:58:29.648 INFO:tasks.rados.rados.0.smithi079.stdout:748: oid 19 version 245 is already newer than 245 2024-04-24T01:58:29.648 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 19 v 245 (ObjNum 361 snap 89 seq_num 361) dirty exists 2024-04-24T01:58:29.648 INFO:tasks.rados.rados.0.smithi079.stdout:748: left oid 19 (ObjNum 361 snap 89 seq_num 361) 2024-04-24T01:58:29.656 INFO:tasks.rados.rados.0.smithi079.stdout:748: done (4 left) 2024-04-24T01:58:29.656 INFO:tasks.rados.rados.0.smithi079.stdout:749: done (3 left) 2024-04-24T01:58:29.656 INFO:tasks.rados.rados.0.smithi079.stdout:752: done (2 left) 2024-04-24T01:58:29.656 INFO:tasks.rados.rados.0.smithi079.stdout:753: delete oid 45 current snap is 89 2024-04-24T01:58:29.664 INFO:tasks.rados.rados.0.smithi079.stdout:750: finishing write tid 2 to smithi07929762-25 2024-04-24T01:58:29.682 INFO:tasks.rados.rados.0.smithi079.stdout:751: finishing write tid 1 to smithi07929762-44 2024-04-24T01:58:29.687 INFO:tasks.rados.rados.0.smithi079.stdout:750: finishing write tid 3 to smithi07929762-25 2024-04-24T01:58:29.688 INFO:tasks.rados.rados.0.smithi079.stdout:750: finishing write tid 4 to smithi07929762-25 2024-04-24T01:58:29.688 INFO:tasks.rados.rados.0.smithi079.stdout:750: oid 25 updating version 0 to 293 2024-04-24T01:58:29.688 INFO:tasks.rados.rados.0.smithi079.stdout:750: oid 25 version 293 is already newer than 293 2024-04-24T01:58:29.689 INFO:tasks.rados.rados.0.smithi079.stdout:750: oid 25 version 293 is already newer than 293 2024-04-24T01:58:29.689 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 25 v 293 (ObjNum 362 snap 89 seq_num 362) dirty exists 2024-04-24T01:58:29.689 INFO:tasks.rados.rados.0.smithi079.stdout:750: left oid 25 (ObjNum 362 snap 89 seq_num 362) 2024-04-24T01:58:29.705 INFO:tasks.rados.rados.0.smithi079.stdout:750: done (2 left) 2024-04-24T01:58:29.706 INFO:tasks.rados.rados.0.smithi079.stdout:753: done (1 left) 2024-04-24T01:58:29.706 INFO:tasks.rados.rados.0.smithi079.stdout:754: snap_create 2024-04-24T01:58:29.706 INFO:tasks.rados.rados.0.smithi079.stdout:751: finishing write tid 2 to smithi07929762-44 2024-04-24T01:58:29.716 INFO:tasks.rados.rados.0.smithi079.stdout:751: finishing write tid 3 to smithi07929762-44 2024-04-24T01:58:29.718 INFO:tasks.rados.rados.0.smithi079.stdout:751: finishing write tid 4 to smithi07929762-44 2024-04-24T01:58:29.718 INFO:tasks.rados.rados.0.smithi079.stdout:751: oid 44 updating version 0 to 196 2024-04-24T01:58:29.718 INFO:tasks.rados.rados.0.smithi079.stdout:751: oid 44 updating version 196 to 198 2024-04-24T01:58:29.718 INFO:tasks.rados.rados.0.smithi079.stdout:751: oid 44 version 198 is already newer than 198 2024-04-24T01:58:29.718 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 44 v 198 (ObjNum 363 snap 89 seq_num 363) dirty exists 2024-04-24T01:58:29.718 INFO:tasks.rados.rados.0.smithi079.stdout:751: left oid 44 (ObjNum 363 snap 89 seq_num 363) 2024-04-24T01:58:30.044 INFO:tasks.rados.rados.0.smithi079.stdout:751: done (1 left) 2024-04-24T01:58:30.044 INFO:tasks.rados.rados.0.smithi079.stdout:754: done (0 left) 2024-04-24T01:58:30.044 INFO:tasks.rados.rados.0.smithi079.stdout:755: snap_remove snap 88 2024-04-24T01:58:30.920 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T01:58:30.957 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-04-24T01:58:30.958 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-04-24T01:58:30.958 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-04-24T01:58:30.958 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-04-24T01:58:31.029 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:58:31.027+0000 7f681e46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:31.033 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:58:31.033+0000 7f681e46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:31.047 INFO:tasks.rados.rados.0.smithi079.stdout:755: done (0 left) 2024-04-24T01:58:31.047 INFO:tasks.rados.rados.0.smithi079.stdout:756: snap_remove snap 89 2024-04-24T01:58:31.082 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:58:31.074+0000 7ff0e526d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:31.086 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:58:31.086+0000 7ff0e526d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:31.102 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T01:58:31.102 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T01:58:31.102 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T01:58:31.102 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T01:58:31.102 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T01:58:31.102 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T01:58:31.102 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T01:58:31.102 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T01:58:31.103 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T01:58:31.103 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T01:58:31.103 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T01:58:31.103 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T01:58:31.105 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T01:58:31.105 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T01:58:31.106 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-04-24T01:58:31.235 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:58:31.230+0000 7f393155d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:31.238 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:58:31.238+0000 7f393155d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:31.252 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T01:58:31.252 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T01:58:31.253 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T01:58:31.253 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T01:58:31.253 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T01:58:31.253 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T01:58:31.253 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T01:58:31.253 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T01:58:31.253 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T01:58:31.253 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T01:58:31.253 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T01:58:31.253 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T01:58:31.255 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T01:58:31.255 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T01:58:31.256 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-04-24T01:58:31.363 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:58:31.358+0000 7f8e676ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:31.367 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:58:31.367+0000 7f8e676ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:31.382 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T01:58:31.382 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T01:58:31.382 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T01:58:31.382 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T01:58:31.382 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T01:58:31.382 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T01:58:31.382 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T01:58:31.382 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T01:58:31.382 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T01:58:31.382 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T01:58:31.382 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T01:58:31.382 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T01:58:31.385 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T01:58:31.385 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T01:58:31.385 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-04-24T01:58:31.493 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:58:31.489+0000 7f666a42d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:31.497 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:58:31.497+0000 7f666a42d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:31.513 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T01:58:31.513 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T01:58:31.514 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T01:58:31.514 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T01:58:31.514 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T01:58:31.514 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T01:58:31.514 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T01:58:31.514 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T01:58:31.514 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T01:58:31.514 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T01:58:31.514 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T01:58:31.514 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T01:58:31.516 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T01:58:31.517 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T01:58:32.051 INFO:tasks.rados.rados.0.smithi079.stdout:756: done (0 left) 2024-04-24T01:58:32.051 INFO:tasks.rados.rados.0.smithi079.stdout:757: write oid 26 current snap is 90 2024-04-24T01:58:32.051 INFO:tasks.rados.rados.0.smithi079.stdout:757: seq_num 366 ranges {551362=726078,1717586=409779} 2024-04-24T01:58:32.058 INFO:tasks.rados.rados.0.smithi079.stdout:757: writing smithi07929762-26 from 551362 to 1277440 tid 1 2024-04-24T01:58:32.064 INFO:tasks.rados.rados.0.smithi079.stdout:757: writing smithi07929762-26 from 1717586 to 2127365 tid 2 2024-04-24T01:58:32.064 INFO:tasks.rados.rados.0.smithi079.stdout:758: read oid 29 snap -1 2024-04-24T01:58:32.064 INFO:tasks.rados.rados.0.smithi079.stdout:758: expect (ObjNum 294 snap 78 seq_num 294) 2024-04-24T01:58:32.064 INFO:tasks.rados.rados.0.smithi079.stdout:759: write (excl) oid 9 current snap is 90 2024-04-24T01:58:32.064 INFO:tasks.rados.rados.0.smithi079.stdout:759: seq_num 367 ranges {568169=401327,1376068=669790} 2024-04-24T01:58:32.070 INFO:tasks.rados.rados.0.smithi079.stdout:759: writing smithi07929762-9 from 568169 to 969496 tid 1 2024-04-24T01:58:32.079 INFO:tasks.rados.rados.0.smithi079.stdout:759: writing smithi07929762-9 from 1376068 to 2045858 tid 2 2024-04-24T01:58:32.079 INFO:tasks.rados.rados.0.smithi079.stdout:760: snap_remove snap 87 2024-04-24T01:58:33.057 INFO:tasks.rados.rados.0.smithi079.stdout:760: done (3 left) 2024-04-24T01:58:33.057 INFO:tasks.rados.rados.0.smithi079.stdout:761: delete oid 5 current snap is 90 2024-04-24T01:58:33.057 INFO:tasks.rados.rados.0.smithi079.stdout:758: expect (ObjNum 294 snap 78 seq_num 294) 2024-04-24T01:58:33.141 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T01:58:33.227 INFO:tasks.rados.rados.0.smithi079.stdout:759: finishing write tid 1 to smithi07929762-9 2024-04-24T01:58:33.227 INFO:tasks.rados.rados.0.smithi079.stdout:758: done (3 left) 2024-04-24T01:58:33.228 INFO:tasks.rados.rados.0.smithi079.stdout:761: done (2 left) 2024-04-24T01:58:33.228 INFO:tasks.rados.rados.0.smithi079.stdout:762: write (excl) oid 40 current snap is 90 2024-04-24T01:58:33.228 INFO:tasks.rados.rados.0.smithi079.stdout:757: finishing write tid 1 to smithi07929762-26 2024-04-24T01:58:33.228 INFO:tasks.rados.rados.0.smithi079.stdout:757: finishing write tid 2 to smithi07929762-26 2024-04-24T01:58:33.228 INFO:tasks.rados.rados.0.smithi079.stdout:762: seq_num 369 ranges {601783=551825,1776679=425104} 2024-04-24T01:58:33.233 INFO:tasks.rados.rados.0.smithi079.stdout:762: writing smithi07929762-40 from 601783 to 1153608 tid 1 2024-04-24T01:58:33.240 INFO:tasks.rados.rados.0.smithi079.stdout:762: writing smithi07929762-40 from 1776679 to 2201783 tid 2 2024-04-24T01:58:33.240 INFO:tasks.rados.rados.0.smithi079.stdout:763: write oid 41 current snap is 90 2024-04-24T01:58:33.240 INFO:tasks.rados.rados.0.smithi079.stdout:759: finishing write tid 2 to smithi07929762-9 2024-04-24T01:58:33.240 INFO:tasks.rados.rados.0.smithi079.stdout:763: seq_num 370 ranges {618590=627074,1835161=383429} 2024-04-24T01:58:33.247 INFO:tasks.rados.rados.0.smithi079.stdout:763: writing smithi07929762-41 from 618590 to 1245664 tid 1 2024-04-24T01:58:33.249 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:58:33.246+0000 7fb495024640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:33.253 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:58:33.252+0000 7fb495024640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:33.254 INFO:tasks.rados.rados.0.smithi079.stdout:763: writing smithi07929762-41 from 1835161 to 2218590 tid 2 2024-04-24T01:58:33.254 INFO:tasks.rados.rados.0.smithi079.stdout:764: write (excl) oid 45 current snap is 90 2024-04-24T01:58:33.254 INFO:tasks.rados.rados.0.smithi079.stdout:757: finishing write tid 3 to smithi07929762-26 2024-04-24T01:58:33.254 INFO:tasks.rados.rados.0.smithi079.stdout:757: finishing write tid 4 to smithi07929762-26 2024-04-24T01:58:33.254 INFO:tasks.rados.rados.0.smithi079.stdout:757: oid 26 updating version 0 to 298 2024-04-24T01:58:33.254 INFO:tasks.rados.rados.0.smithi079.stdout:757: oid 26 version 298 is already newer than 298 2024-04-24T01:58:33.254 INFO:tasks.rados.rados.0.smithi079.stdout:757: oid 26 version 298 is already newer than 298 2024-04-24T01:58:33.255 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 26 v 298 (ObjNum 366 snap 90 seq_num 366) dirty exists 2024-04-24T01:58:33.255 INFO:tasks.rados.rados.0.smithi079.stdout:757: left oid 26 (ObjNum 366 snap 90 seq_num 366) 2024-04-24T01:58:33.255 INFO:tasks.rados.rados.0.smithi079.stdout:759: finishing write tid 3 to smithi07929762-9 2024-04-24T01:58:33.255 INFO:tasks.rados.rados.0.smithi079.stdout:764: seq_num 371 ranges {635397=702323,1893643=341754} 2024-04-24T01:58:33.263 INFO:tasks.rados.rados.0.smithi079.stdout:764: writing smithi07929762-45 from 635397 to 1337720 tid 1 2024-04-24T01:58:33.269 INFO:tasks.rados.rados.0.smithi079.stdout:764: writing smithi07929762-45 from 1893643 to 2235397 tid 2 2024-04-24T01:58:33.269 INFO:tasks.rados.rados.0.smithi079.stdout:757: done (4 left) 2024-04-24T01:58:33.269 INFO:tasks.rados.rados.0.smithi079.stdout:765: write (excl) oid 50 current snap is 90 2024-04-24T01:58:33.269 INFO:tasks.rados.rados.0.smithi079.stdout:765: seq_num 372 ranges {652204=777572,1952125=300079} 2024-04-24T01:58:33.279 INFO:tasks.rados.rados.0.smithi079.stdout:765: writing smithi07929762-50 from 652204 to 1429776 tid 1 2024-04-24T01:58:33.285 INFO:tasks.rados.rados.0.smithi079.stdout:765: writing smithi07929762-50 from 1952125 to 2252204 tid 2 2024-04-24T01:58:33.285 INFO:tasks.rados.rados.0.smithi079.stdout:766: write (excl) oid 44 current snap is 90 2024-04-24T01:58:33.286 INFO:tasks.rados.rados.0.smithi079.stdout:766: seq_num 373 ranges {669011=569174,2010607=258404} 2024-04-24T01:58:33.295 INFO:tasks.rados.rados.0.smithi079.stdout:766: writing smithi07929762-44 from 669011 to 1238185 tid 1 2024-04-24T01:58:33.303 INFO:tasks.rados.rados.0.smithi079.stdout:766: writing smithi07929762-44 from 2010607 to 2269011 tid 2 2024-04-24T01:58:33.303 INFO:tasks.rados.rados.0.smithi079.stdout:767: write (excl) oid 32 current snap is 90 2024-04-24T01:58:33.303 INFO:tasks.rados.rados.0.smithi079.stdout:759: finishing write tid 4 to smithi07929762-9 2024-04-24T01:58:33.303 INFO:tasks.rados.rados.0.smithi079.stdout:759: oid 9 updating version 0 to 263 2024-04-24T01:58:33.303 INFO:tasks.rados.rados.0.smithi079.stdout:759: oid 9 updating version 263 to 265 2024-04-24T01:58:33.303 INFO:tasks.rados.rados.0.smithi079.stdout:759: oid 9 version 265 is already newer than 265 2024-04-24T01:58:33.303 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 9 v 265 (ObjNum 367 snap 90 seq_num 367) dirty exists 2024-04-24T01:58:33.303 INFO:tasks.rados.rados.0.smithi079.stdout:759: left oid 9 (ObjNum 367 snap 90 seq_num 367) 2024-04-24T01:58:33.303 INFO:tasks.rados.rados.0.smithi079.stdout:767: seq_num 374 ranges {685818=644423,2069089=216729} 2024-04-24T01:58:33.310 INFO:tasks.rados.rados.0.smithi079.stdout:767: writing smithi07929762-32 from 685818 to 1330241 tid 1 2024-04-24T01:58:33.317 INFO:tasks.rados.rados.0.smithi079.stdout:767: writing smithi07929762-32 from 2069089 to 2285818 tid 2 2024-04-24T01:58:33.317 INFO:tasks.rados.rados.0.smithi079.stdout:759: done (6 left) 2024-04-24T01:58:33.317 INFO:tasks.rados.rados.0.smithi079.stdout:768: write (excl) oid 7 current snap is 90 2024-04-24T01:58:33.317 INFO:tasks.rados.rados.0.smithi079.stdout:768: seq_num 375 ranges {702625=719672,2127571=175054} 2024-04-24T01:58:33.328 INFO:tasks.rados.rados.0.smithi079.stdout:768: writing smithi07929762-7 from 702625 to 1422297 tid 1 2024-04-24T01:58:33.334 INFO:tasks.rados.rados.0.smithi079.stdout:768: writing smithi07929762-7 from 2127571 to 2302625 tid 2 2024-04-24T01:58:33.334 INFO:tasks.rados.rados.0.smithi079.stdout:769: read oid 48 snap 83 2024-04-24T01:58:33.334 INFO:tasks.rados.rados.0.smithi079.stdout:769: expect (ObjNum 284 snap 75 seq_num 284) 2024-04-24T01:58:33.334 INFO:tasks.rados.rados.0.smithi079.stdout:770: snap_remove snap 86 2024-04-24T01:58:34.058 INFO:tasks.rados.rados.0.smithi079.stdout:770: done (8 left) 2024-04-24T01:58:34.058 INFO:tasks.rados.rados.0.smithi079.stdout:771: read oid 46 snap 83 2024-04-24T01:58:34.058 INFO:tasks.rados.rados.0.smithi079.stdout:771: expect deleted 2024-04-24T01:58:34.058 INFO:tasks.rados.rados.0.smithi079.stdout:772: read oid 4 snap -1 2024-04-24T01:58:34.058 INFO:tasks.rados.rados.0.smithi079.stdout:772: expect deleted 2024-04-24T01:58:34.058 INFO:tasks.rados.rados.0.smithi079.stdout:773: write (excl) oid 24 current snap is 90 2024-04-24T01:58:34.059 INFO:tasks.rados.rados.0.smithi079.stdout:773: seq_num 376 ranges {719432=794921,2186053=133379} 2024-04-24T01:58:34.066 INFO:tasks.rados.rados.0.smithi079.stdout:773: writing smithi07929762-24 from 719432 to 1514353 tid 1 2024-04-24T01:58:34.071 INFO:tasks.rados.rados.0.smithi079.stdout:773: writing smithi07929762-24 from 2186053 to 2319432 tid 2 2024-04-24T01:58:34.071 INFO:tasks.rados.rados.0.smithi079.stdout:774: write oid 12 current snap is 90 2024-04-24T01:58:34.071 INFO:tasks.rados.rados.0.smithi079.stdout:774: seq_num 377 ranges {736239=470170,1728182=571782} 2024-04-24T01:58:34.077 INFO:tasks.rados.rados.0.smithi079.stdout:774: writing smithi07929762-12 from 736239 to 1206409 tid 1 2024-04-24T01:58:34.085 INFO:tasks.rados.rados.0.smithi079.stdout:774: writing smithi07929762-12 from 1728182 to 2299964 tid 2 2024-04-24T01:58:34.085 INFO:tasks.rados.rados.0.smithi079.stdout:775: snap_create 2024-04-24T01:58:34.085 INFO:tasks.rados.rados.0.smithi079.stdout:764: finishing write tid 1 to smithi07929762-45 2024-04-24T01:58:34.085 INFO:tasks.rados.rados.0.smithi079.stdout:765: finishing write tid 1 to smithi07929762-50 2024-04-24T01:58:34.085 INFO:tasks.rados.rados.0.smithi079.stdout:766: finishing write tid 1 to smithi07929762-44 2024-04-24T01:58:34.085 INFO:tasks.rados.rados.0.smithi079.stdout:762: finishing write tid 1 to smithi07929762-40 2024-04-24T01:58:34.085 INFO:tasks.rados.rados.0.smithi079.stdout:764: finishing write tid 2 to smithi07929762-45 2024-04-24T01:58:34.085 INFO:tasks.rados.rados.0.smithi079.stdout:765: finishing write tid 2 to smithi07929762-50 2024-04-24T01:58:34.085 INFO:tasks.rados.rados.0.smithi079.stdout:767: finishing write tid 1 to smithi07929762-32 2024-04-24T01:58:34.085 INFO:tasks.rados.rados.0.smithi079.stdout:764: finishing write tid 3 to smithi07929762-45 2024-04-24T01:58:34.085 INFO:tasks.rados.rados.0.smithi079.stdout:763: finishing write tid 1 to smithi07929762-41 2024-04-24T01:58:34.085 INFO:tasks.rados.rados.0.smithi079.stdout:764: finishing write tid 4 to smithi07929762-45 2024-04-24T01:58:34.086 INFO:tasks.rados.rados.0.smithi079.stdout:764: oid 45 updating version 0 to 231 2024-04-24T01:58:34.086 INFO:tasks.rados.rados.0.smithi079.stdout:764: oid 45 updating version 231 to 233 2024-04-24T01:58:34.086 INFO:tasks.rados.rados.0.smithi079.stdout:764: oid 45 version 233 is already newer than 233 2024-04-24T01:58:34.086 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 45 v 233 (ObjNum 371 snap 90 seq_num 371) dirty exists 2024-04-24T01:58:34.086 INFO:tasks.rados.rados.0.smithi079.stdout:764: left oid 45 (ObjNum 371 snap 90 seq_num 371) 2024-04-24T01:58:34.086 INFO:tasks.rados.rados.0.smithi079.stdout:765: finishing write tid 3 to smithi07929762-50 2024-04-24T01:58:34.086 INFO:tasks.rados.rados.0.smithi079.stdout:763: finishing write tid 2 to smithi07929762-41 2024-04-24T01:58:34.086 INFO:tasks.rados.rados.0.smithi079.stdout:763: finishing write tid 3 to smithi07929762-41 2024-04-24T01:58:34.086 INFO:tasks.rados.rados.0.smithi079.stdout:763: finishing write tid 4 to smithi07929762-41 2024-04-24T01:58:34.086 INFO:tasks.rados.rados.0.smithi079.stdout:763: oid 41 updating version 0 to 127 2024-04-24T01:58:34.086 INFO:tasks.rados.rados.0.smithi079.stdout:763: oid 41 version 127 is already newer than 127 2024-04-24T01:58:34.086 INFO:tasks.rados.rados.0.smithi079.stdout:763: oid 41 version 127 is already newer than 127 2024-04-24T01:58:34.086 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 41 v 127 (ObjNum 370 snap 90 seq_num 370) dirty exists 2024-04-24T01:58:34.086 INFO:tasks.rados.rados.0.smithi079.stdout:763: left oid 41 (ObjNum 370 snap 90 seq_num 370) 2024-04-24T01:58:34.086 INFO:tasks.rados.rados.0.smithi079.stdout:765: finishing write tid 4 to smithi07929762-50 2024-04-24T01:58:34.087 INFO:tasks.rados.rados.0.smithi079.stdout:765: oid 50 updating version 0 to 208 2024-04-24T01:58:34.087 INFO:tasks.rados.rados.0.smithi079.stdout:765: oid 50 version 208 is already newer than 208 2024-04-24T01:58:34.087 INFO:tasks.rados.rados.0.smithi079.stdout:765: oid 50 version 208 is already newer than 206 2024-04-24T01:58:34.087 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 50 v 208 (ObjNum 372 snap 90 seq_num 372) dirty exists 2024-04-24T01:58:34.087 INFO:tasks.rados.rados.0.smithi079.stdout:765: left oid 50 (ObjNum 372 snap 90 seq_num 372) 2024-04-24T01:58:34.087 INFO:tasks.rados.rados.0.smithi079.stdout:766: finishing write tid 2 to smithi07929762-44 2024-04-24T01:58:34.087 INFO:tasks.rados.rados.0.smithi079.stdout:767: finishing write tid 2 to smithi07929762-32 2024-04-24T01:58:34.087 INFO:tasks.rados.rados.0.smithi079.stdout:766: finishing write tid 3 to smithi07929762-44 2024-04-24T01:58:34.087 INFO:tasks.rados.rados.0.smithi079.stdout:766: finishing write tid 4 to smithi07929762-44 2024-04-24T01:58:34.087 INFO:tasks.rados.rados.0.smithi079.stdout:766: oid 44 updating version 0 to 205 2024-04-24T01:58:34.087 INFO:tasks.rados.rados.0.smithi079.stdout:766: oid 44 version 205 is already newer than 205 2024-04-24T01:58:34.087 INFO:tasks.rados.rados.0.smithi079.stdout:766: oid 44 version 205 is already newer than 203 2024-04-24T01:58:34.087 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 44 v 205 (ObjNum 373 snap 90 seq_num 373) dirty exists 2024-04-24T01:58:34.087 INFO:tasks.rados.rados.0.smithi079.stdout:766: left oid 44 (ObjNum 373 snap 90 seq_num 373) 2024-04-24T01:58:34.088 INFO:tasks.rados.rados.0.smithi079.stdout:767: finishing write tid 3 to smithi07929762-32 2024-04-24T01:58:34.088 INFO:tasks.rados.rados.0.smithi079.stdout:767: finishing write tid 4 to smithi07929762-32 2024-04-24T01:58:34.088 INFO:tasks.rados.rados.0.smithi079.stdout:767: oid 32 updating version 0 to 131 2024-04-24T01:58:34.088 INFO:tasks.rados.rados.0.smithi079.stdout:767: oid 32 version 131 is already newer than 131 2024-04-24T01:58:34.088 INFO:tasks.rados.rados.0.smithi079.stdout:767: oid 32 version 131 is already newer than 129 2024-04-24T01:58:34.088 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 32 v 131 (ObjNum 374 snap 90 seq_num 374) dirty exists 2024-04-24T01:58:34.088 INFO:tasks.rados.rados.0.smithi079.stdout:767: left oid 32 (ObjNum 374 snap 90 seq_num 374) 2024-04-24T01:58:34.088 INFO:tasks.rados.rados.0.smithi079.stdout:762: finishing write tid 2 to smithi07929762-40 2024-04-24T01:58:34.088 INFO:tasks.rados.rados.0.smithi079.stdout:762: finishing write tid 3 to smithi07929762-40 2024-04-24T01:58:34.088 INFO:tasks.rados.rados.0.smithi079.stdout:762: finishing write tid 4 to smithi07929762-40 2024-04-24T01:58:34.088 INFO:tasks.rados.rados.0.smithi079.stdout:762: oid 40 updating version 0 to 270 2024-04-24T01:58:34.088 INFO:tasks.rados.rados.0.smithi079.stdout:762: oid 40 version 270 is already newer than 268 2024-04-24T01:58:34.088 INFO:tasks.rados.rados.0.smithi079.stdout:762: oid 40 version 270 is already newer than 270 2024-04-24T01:58:34.088 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 40 v 270 (ObjNum 369 snap 90 seq_num 369) dirty exists 2024-04-24T01:58:34.088 INFO:tasks.rados.rados.0.smithi079.stdout:762: left oid 40 (ObjNum 369 snap 90 seq_num 369) 2024-04-24T01:58:34.088 INFO:tasks.rados.rados.0.smithi079.stdout:769: expect (ObjNum 284 snap 75 seq_num 284) 2024-04-24T01:58:34.209 INFO:tasks.rados.rados.0.smithi079.stdout:768: finishing write tid 1 to smithi07929762-7 2024-04-24T01:58:34.210 INFO:tasks.rados.rados.0.smithi079.stdout:768: finishing write tid 2 to smithi07929762-7 2024-04-24T01:58:34.210 INFO:tasks.rados.rados.0.smithi079.stdout:768: finishing write tid 3 to smithi07929762-7 2024-04-24T01:58:34.210 INFO:tasks.rados.rados.0.smithi079.stdout:768: finishing write tid 4 to smithi07929762-7 2024-04-24T01:58:34.210 INFO:tasks.rados.rados.0.smithi079.stdout:768: oid 7 updating version 0 to 208 2024-04-24T01:58:34.210 INFO:tasks.rados.rados.0.smithi079.stdout:768: oid 7 version 208 is already newer than 208 2024-04-24T01:58:34.210 INFO:tasks.rados.rados.0.smithi079.stdout:768: oid 7 version 208 is already newer than 206 2024-04-24T01:58:34.210 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 7 v 208 (ObjNum 375 snap 90 seq_num 375) dirty exists 2024-04-24T01:58:34.210 INFO:tasks.rados.rados.0.smithi079.stdout:768: left oid 7 (ObjNum 375 snap 90 seq_num 375) 2024-04-24T01:58:34.210 INFO:tasks.rados.rados.0.smithi079.stdout:774: finishing write tid 1 to smithi07929762-12 2024-04-24T01:58:34.210 INFO:tasks.rados.rados.0.smithi079.stdout:773: finishing write tid 1 to smithi07929762-24 2024-04-24T01:58:34.210 INFO:tasks.rados.rados.0.smithi079.stdout:773: finishing write tid 2 to smithi07929762-24 2024-04-24T01:58:34.211 INFO:tasks.rados.rados.0.smithi079.stdout:774: finishing write tid 2 to smithi07929762-12 2024-04-24T01:58:34.211 INFO:tasks.rados.rados.0.smithi079.stdout:773: finishing write tid 3 to smithi07929762-24 2024-04-24T01:58:34.211 INFO:tasks.rados.rados.0.smithi079.stdout:773: finishing write tid 4 to smithi07929762-24 2024-04-24T01:58:34.211 INFO:tasks.rados.rados.0.smithi079.stdout:773: oid 24 updating version 0 to 260 2024-04-24T01:58:34.211 INFO:tasks.rados.rados.0.smithi079.stdout:773: oid 24 version 260 is already newer than 258 2024-04-24T01:58:34.211 INFO:tasks.rados.rados.0.smithi079.stdout:773: oid 24 version 260 is already newer than 260 2024-04-24T01:58:34.211 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 24 v 260 (ObjNum 376 snap 90 seq_num 376) dirty exists 2024-04-24T01:58:34.211 INFO:tasks.rados.rados.0.smithi079.stdout:773: left oid 24 (ObjNum 376 snap 90 seq_num 376) 2024-04-24T01:58:34.211 INFO:tasks.rados.rados.0.smithi079.stdout:774: finishing write tid 3 to smithi07929762-12 2024-04-24T01:58:34.212 INFO:tasks.rados.rados.0.smithi079.stdout:774: finishing write tid 4 to smithi07929762-12 2024-04-24T01:58:34.212 INFO:tasks.rados.rados.0.smithi079.stdout:774: oid 12 updating version 0 to 205 2024-04-24T01:58:34.212 INFO:tasks.rados.rados.0.smithi079.stdout:774: oid 12 version 205 is already newer than 205 2024-04-24T01:58:34.212 INFO:tasks.rados.rados.0.smithi079.stdout:774: oid 12 version 205 is already newer than 205 2024-04-24T01:58:34.212 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 12 v 205 (ObjNum 377 snap 90 seq_num 377) dirty exists 2024-04-24T01:58:34.212 INFO:tasks.rados.rados.0.smithi079.stdout:774: left oid 12 (ObjNum 377 snap 90 seq_num 377) 2024-04-24T01:58:35.063 INFO:tasks.rados.rados.0.smithi079.stdout:762: done (12 left) 2024-04-24T01:58:35.064 INFO:tasks.rados.rados.0.smithi079.stdout:763: done (11 left) 2024-04-24T01:58:35.064 INFO:tasks.rados.rados.0.smithi079.stdout:764: done (10 left) 2024-04-24T01:58:35.064 INFO:tasks.rados.rados.0.smithi079.stdout:765: done (9 left) 2024-04-24T01:58:35.064 INFO:tasks.rados.rados.0.smithi079.stdout:766: done (8 left) 2024-04-24T01:58:35.064 INFO:tasks.rados.rados.0.smithi079.stdout:767: done (7 left) 2024-04-24T01:58:35.064 INFO:tasks.rados.rados.0.smithi079.stdout:768: done (6 left) 2024-04-24T01:58:35.064 INFO:tasks.rados.rados.0.smithi079.stdout:769: done (5 left) 2024-04-24T01:58:35.064 INFO:tasks.rados.rados.0.smithi079.stdout:771: done (4 left) 2024-04-24T01:58:35.064 INFO:tasks.rados.rados.0.smithi079.stdout:772: done (3 left) 2024-04-24T01:58:35.064 INFO:tasks.rados.rados.0.smithi079.stdout:773: done (2 left) 2024-04-24T01:58:35.064 INFO:tasks.rados.rados.0.smithi079.stdout:774: done (1 left) 2024-04-24T01:58:35.064 INFO:tasks.rados.rados.0.smithi079.stdout:775: done (0 left) 2024-04-24T01:58:35.064 INFO:tasks.rados.rados.0.smithi079.stdout:776: delete oid 31 current snap is 91 2024-04-24T01:58:35.108 INFO:tasks.rados.rados.0.smithi079.stdout:776: done (0 left) 2024-04-24T01:58:35.108 INFO:tasks.rados.rados.0.smithi079.stdout:777: snap_create 2024-04-24T01:58:35.508 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T01:58:35.615 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:58:35.612+0000 7f051d017640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:35.619 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:58:35.618+0000 7f051d017640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:36.039 INFO:tasks.rados.rados.0.smithi079.stdout:777: done (0 left) 2024-04-24T01:58:36.039 INFO:tasks.rados.rados.0.smithi079.stdout:778: write oid 17 current snap is 92 2024-04-24T01:58:36.040 INFO:tasks.rados.rados.0.smithi079.stdout:778: seq_num 379 ranges {769853=620668,1845146=524707} 2024-04-24T01:58:36.048 INFO:tasks.rados.rados.0.smithi079.stdout:778: writing smithi07929762-17 from 769853 to 1390521 tid 1 2024-04-24T01:58:36.054 INFO:tasks.rados.rados.0.smithi079.stdout:778: writing smithi07929762-17 from 1845146 to 2369853 tid 2 2024-04-24T01:58:36.054 INFO:tasks.rados.rados.0.smithi079.stdout:779: write oid 21 current snap is 92 2024-04-24T01:58:36.054 INFO:tasks.rados.rados.0.smithi079.stdout:779: seq_num 380 ranges {786660=695917,1903628=435462} 2024-04-24T01:58:36.065 INFO:tasks.rados.rados.0.smithi079.stdout:779: writing smithi07929762-21 from 786660 to 1482577 tid 1 2024-04-24T01:58:36.071 INFO:tasks.rados.rados.0.smithi079.stdout:779: writing smithi07929762-21 from 1903628 to 2339090 tid 2 2024-04-24T01:58:36.071 INFO:tasks.rados.rados.0.smithi079.stdout:780: write (excl) oid 16 current snap is 92 2024-04-24T01:58:36.071 INFO:tasks.rados.rados.0.smithi079.stdout:780: seq_num 381 ranges {403467=487519,1562110=579120} 2024-04-24T01:58:36.076 INFO:tasks.rados.rados.0.smithi079.stdout:780: writing smithi07929762-16 from 403467 to 890986 tid 1 2024-04-24T01:58:36.085 INFO:tasks.rados.rados.0.smithi079.stdout:780: writing smithi07929762-16 from 1562110 to 2141230 tid 2 2024-04-24T01:58:36.085 INFO:tasks.rados.rados.0.smithi079.stdout:781: write (excl) oid 25 current snap is 92 2024-04-24T01:58:36.085 INFO:tasks.rados.rados.0.smithi079.stdout:781: seq_num 382 ranges {420274=562768,1620592=439131} 2024-04-24T01:58:36.090 INFO:tasks.rados.rados.0.smithi079.stdout:781: writing smithi07929762-25 from 420274 to 983042 tid 1 2024-04-24T01:58:36.096 INFO:tasks.rados.rados.0.smithi079.stdout:781: writing smithi07929762-25 from 1620592 to 2059723 tid 2 2024-04-24T01:58:36.096 INFO:tasks.rados.rados.0.smithi079.stdout:782: write oid 47 current snap is 92 2024-04-24T01:58:36.097 INFO:tasks.rados.rados.0.smithi079.stdout:782: seq_num 383 ranges {437081=638017,1679074=582789} 2024-04-24T01:58:36.103 INFO:tasks.rados.rados.0.smithi079.stdout:782: writing smithi07929762-47 from 437081 to 1075098 tid 1 2024-04-24T01:58:36.114 INFO:tasks.rados.rados.0.smithi079.stdout:782: writing smithi07929762-47 from 1679074 to 2261863 tid 2 2024-04-24T01:58:36.114 INFO:tasks.rados.rados.0.smithi079.stdout:783: delete oid 22 current snap is 92 2024-04-24T01:58:36.142 INFO:tasks.rados.rados.0.smithi079.stdout:779: finishing write tid 1 to smithi07929762-21 2024-04-24T01:58:36.144 INFO:tasks.rados.rados.0.smithi079.stdout:778: finishing write tid 1 to smithi07929762-17 2024-04-24T01:58:36.166 INFO:tasks.rados.rados.0.smithi079.stdout:779: finishing write tid 2 to smithi07929762-21 2024-04-24T01:58:36.166 INFO:tasks.rados.rados.0.smithi079.stdout:780: finishing write tid 1 to smithi07929762-16 2024-04-24T01:58:36.166 INFO:tasks.rados.rados.0.smithi079.stdout:778: finishing write tid 2 to smithi07929762-17 2024-04-24T01:58:36.177 INFO:tasks.rados.rados.0.smithi079.stdout:778: finishing write tid 3 to smithi07929762-17 2024-04-24T01:58:36.179 INFO:tasks.rados.rados.0.smithi079.stdout:778: finishing write tid 4 to smithi07929762-17 2024-04-24T01:58:36.179 INFO:tasks.rados.rados.0.smithi079.stdout:778: oid 17 updating version 0 to 208 2024-04-24T01:58:36.179 INFO:tasks.rados.rados.0.smithi079.stdout:778: oid 17 version 208 is already newer than 208 2024-04-24T01:58:36.179 INFO:tasks.rados.rados.0.smithi079.stdout:778: oid 17 version 208 is already newer than 208 2024-04-24T01:58:36.179 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 17 v 208 (ObjNum 379 snap 92 seq_num 379) dirty exists 2024-04-24T01:58:36.180 INFO:tasks.rados.rados.0.smithi079.stdout:778: left oid 17 (ObjNum 379 snap 92 seq_num 379) 2024-04-24T01:58:36.180 INFO:tasks.rados.rados.0.smithi079.stdout:781: finishing write tid 1 to smithi07929762-25 2024-04-24T01:58:36.181 INFO:tasks.rados.rados.0.smithi079.stdout:779: finishing write tid 3 to smithi07929762-21 2024-04-24T01:58:36.186 INFO:tasks.rados.rados.0.smithi079.stdout:779: finishing write tid 4 to smithi07929762-21 2024-04-24T01:58:36.186 INFO:tasks.rados.rados.0.smithi079.stdout:779: oid 21 updating version 0 to 238 2024-04-24T01:58:36.186 INFO:tasks.rados.rados.0.smithi079.stdout:779: oid 21 version 238 is already newer than 238 2024-04-24T01:58:36.186 INFO:tasks.rados.rados.0.smithi079.stdout:779: oid 21 version 238 is already newer than 238 2024-04-24T01:58:36.186 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 21 v 238 (ObjNum 380 snap 92 seq_num 380) dirty exists 2024-04-24T01:58:36.187 INFO:tasks.rados.rados.0.smithi079.stdout:779: left oid 21 (ObjNum 380 snap 92 seq_num 380) 2024-04-24T01:58:36.193 INFO:tasks.rados.rados.0.smithi079.stdout:780: finishing write tid 2 to smithi07929762-16 2024-04-24T01:58:36.199 INFO:tasks.rados.rados.0.smithi079.stdout:780: finishing write tid 3 to smithi07929762-16 2024-04-24T01:58:36.202 INFO:tasks.rados.rados.0.smithi079.stdout:780: finishing write tid 4 to smithi07929762-16 2024-04-24T01:58:36.202 INFO:tasks.rados.rados.0.smithi079.stdout:780: oid 16 updating version 0 to 413 2024-04-24T01:58:36.202 INFO:tasks.rados.rados.0.smithi079.stdout:780: oid 16 version 413 is already newer than 413 2024-04-24T01:58:36.202 INFO:tasks.rados.rados.0.smithi079.stdout:780: oid 16 version 413 is already newer than 411 2024-04-24T01:58:36.202 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 16 v 413 (ObjNum 381 snap 92 seq_num 381) dirty exists 2024-04-24T01:58:36.202 INFO:tasks.rados.rados.0.smithi079.stdout:780: left oid 16 (ObjNum 381 snap 92 seq_num 381) 2024-04-24T01:58:36.217 INFO:tasks.rados.rados.0.smithi079.stdout:778: done (5 left) 2024-04-24T01:58:36.218 INFO:tasks.rados.rados.0.smithi079.stdout:779: done (4 left) 2024-04-24T01:58:36.218 INFO:tasks.rados.rados.0.smithi079.stdout:780: done (3 left) 2024-04-24T01:58:36.218 INFO:tasks.rados.rados.0.smithi079.stdout:783: done (2 left) 2024-04-24T01:58:36.218 INFO:tasks.rados.rados.0.smithi079.stdout:784: delete oid 39 current snap is 92 2024-04-24T01:58:36.221 INFO:tasks.rados.rados.0.smithi079.stdout:781: finishing write tid 2 to smithi07929762-25 2024-04-24T01:58:36.225 INFO:tasks.rados.rados.0.smithi079.stdout:781: finishing write tid 3 to smithi07929762-25 2024-04-24T01:58:36.229 INFO:tasks.rados.rados.0.smithi079.stdout:781: finishing write tid 4 to smithi07929762-25 2024-04-24T01:58:36.229 INFO:tasks.rados.rados.0.smithi079.stdout:781: oid 25 updating version 0 to 306 2024-04-24T01:58:36.229 INFO:tasks.rados.rados.0.smithi079.stdout:781: oid 25 version 306 is already newer than 306 2024-04-24T01:58:36.229 INFO:tasks.rados.rados.0.smithi079.stdout:781: oid 25 version 306 is already newer than 304 2024-04-24T01:58:36.229 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 25 v 306 (ObjNum 382 snap 92 seq_num 382) dirty exists 2024-04-24T01:58:36.229 INFO:tasks.rados.rados.0.smithi079.stdout:781: left oid 25 (ObjNum 382 snap 92 seq_num 382) 2024-04-24T01:58:36.260 INFO:tasks.rados.rados.0.smithi079.stdout:781: done (2 left) 2024-04-24T01:58:36.260 INFO:tasks.rados.rados.0.smithi079.stdout:784: done (1 left) 2024-04-24T01:58:36.261 INFO:tasks.rados.rados.0.smithi079.stdout:785: delete oid 43 current snap is 92 2024-04-24T01:58:36.274 INFO:tasks.rados.rados.0.smithi079.stdout:782: finishing write tid 1 to smithi07929762-47 2024-04-24T01:58:36.296 INFO:tasks.rados.rados.0.smithi079.stdout:782: finishing write tid 2 to smithi07929762-47 2024-04-24T01:58:36.303 INFO:tasks.rados.rados.0.smithi079.stdout:785: done (1 left) 2024-04-24T01:58:36.304 INFO:tasks.rados.rados.0.smithi079.stdout:786: read oid 14 snap 90 2024-04-24T01:58:36.304 INFO:tasks.rados.rados.0.smithi079.stdout:786: expect deleted 2024-04-24T01:58:36.304 INFO:tasks.rados.rados.0.smithi079.stdout:787: read oid 50 snap -1 2024-04-24T01:58:36.304 INFO:tasks.rados.rados.0.smithi079.stdout:787: expect (ObjNum 372 snap 90 seq_num 372) 2024-04-24T01:58:36.304 INFO:tasks.rados.rados.0.smithi079.stdout:788: snap_remove snap 83 2024-04-24T01:58:36.517 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-04-24T01:58:36.518 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-04-24T01:58:36.627 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:58:36.621+0000 7f8817824640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:36.631 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:58:36.630+0000 7f8817824640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:37.064 INFO:teuthology.orchestra.run.smithi079.stderr:reweighted osd.0 to 1 (10000) 2024-04-24T01:58:37.067 INFO:tasks.rados.rados.0.smithi079.stdout:788: done (3 left) 2024-04-24T01:58:37.068 INFO:tasks.rados.rados.0.smithi079.stdout:789: snap_create 2024-04-24T01:58:37.068 INFO:tasks.rados.rados.0.smithi079.stdout:782: finishing write tid 3 to smithi07929762-47 2024-04-24T01:58:37.068 INFO:tasks.rados.rados.0.smithi079.stdout:782: finishing write tid 4 to smithi07929762-47 2024-04-24T01:58:37.068 INFO:tasks.rados.rados.0.smithi079.stdout:782: oid 47 updating version 0 to 310 2024-04-24T01:58:37.068 INFO:tasks.rados.rados.0.smithi079.stdout:782: oid 47 version 310 is already newer than 310 2024-04-24T01:58:37.068 INFO:tasks.rados.rados.0.smithi079.stdout:782: oid 47 version 310 is already newer than 310 2024-04-24T01:58:37.068 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 47 v 310 (ObjNum 383 snap 92 seq_num 383) dirty exists 2024-04-24T01:58:37.068 INFO:tasks.rados.rados.0.smithi079.stdout:782: left oid 47 (ObjNum 383 snap 92 seq_num 383) 2024-04-24T01:58:37.068 INFO:tasks.rados.rados.0.smithi079.stdout:787: expect (ObjNum 372 snap 90 seq_num 372) 2024-04-24T01:58:37.078 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-04-24T01:58:37.222 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:58:37.207+0000 7f995acad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:37.235 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:58:37.234+0000 7f995acad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:37.715 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T01:58:37.819 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:58:37.817+0000 7f5e52a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:37.823 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:58:37.822+0000 7f5e52a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:38.065 INFO:teuthology.orchestra.run.smithi079.stderr:reweighted osd.1 to 1 (10000) 2024-04-24T01:58:38.072 INFO:tasks.rados.rados.0.smithi079.stdout:782: done (3 left) 2024-04-24T01:58:38.072 INFO:tasks.rados.rados.0.smithi079.stdout:786: done (2 left) 2024-04-24T01:58:38.072 INFO:tasks.rados.rados.0.smithi079.stdout:787: done (1 left) 2024-04-24T01:58:38.072 INFO:tasks.rados.rados.0.smithi079.stdout:789: done (0 left) 2024-04-24T01:58:38.072 INFO:tasks.rados.rados.0.smithi079.stdout:790: snap_remove snap 92 2024-04-24T01:58:38.082 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-04-24T01:58:38.192 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:58:38.185+0000 7f7b8ccad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:38.196 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:58:38.195+0000 7f7b8ccad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:39.072 INFO:teuthology.orchestra.run.smithi079.stderr:reweighted osd.2 to 1 (10000) 2024-04-24T01:58:39.075 INFO:tasks.rados.rados.0.smithi079.stdout:790: done (0 left) 2024-04-24T01:58:39.075 INFO:tasks.rados.rados.0.smithi079.stdout:791: write oid 9 current snap is 93 2024-04-24T01:58:39.075 INFO:tasks.rados.rados.0.smithi079.stdout:791: seq_num 387 ranges {504309=539013,1796649=590127} 2024-04-24T01:58:39.081 INFO:tasks.rados.rados.0.smithi079.stdout:791: writing smithi07929762-9 from 504309 to 1043322 tid 1 2024-04-24T01:58:39.090 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-04-24T01:58:39.091 INFO:tasks.rados.rados.0.smithi079.stdout:791: writing smithi07929762-9 from 1796649 to 2386776 tid 2 2024-04-24T01:58:39.092 INFO:tasks.rados.rados.0.smithi079.stdout:792: read oid 33 snap -1 2024-04-24T01:58:39.092 INFO:tasks.rados.rados.0.smithi079.stdout:792: expect (ObjNum 347 snap 82 seq_num 347) 2024-04-24T01:58:39.092 INFO:tasks.rados.rados.0.smithi079.stdout:793: read oid 31 snap -1 2024-04-24T01:58:39.092 INFO:tasks.rados.rados.0.smithi079.stdout:793: expect deleted 2024-04-24T01:58:39.092 INFO:tasks.rados.rados.0.smithi079.stdout:794: write oid 29 current snap is 93 2024-04-24T01:58:39.092 INFO:tasks.rados.rados.0.smithi079.stdout:794: seq_num 388 ranges {521116=730615,1971484=549632} 2024-04-24T01:58:39.099 INFO:tasks.rados.rados.0.smithi079.stdout:794: writing smithi07929762-29 from 521116 to 1251731 tid 1 2024-04-24T01:58:39.107 INFO:tasks.rados.rados.0.smithi079.stdout:794: writing smithi07929762-29 from 1971484 to 2521116 tid 2 2024-04-24T01:58:39.107 INFO:tasks.rados.rados.0.smithi079.stdout:795: delete oid 48 current snap is 93 2024-04-24T01:58:39.201 INFO:tasks.rados.rados.0.smithi079.stdout:794: finishing write tid 1 to smithi07929762-29 2024-04-24T01:58:39.224 INFO:tasks.rados.rados.0.smithi079.stdout:793: done (4 left) 2024-04-24T01:58:39.224 INFO:tasks.rados.rados.0.smithi079.stdout:795: done (3 left) 2024-04-24T01:58:39.224 INFO:tasks.rados.rados.0.smithi079.stdout:796: snap_remove snap 90 2024-04-24T01:58:39.263 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:58:39.252+0000 7f5f97b1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:39.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:58:39.267+0000 7f5f97b1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:39.917 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T01:58:40.024 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:58:40.021+0000 7f093a0ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:40.027 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:58:40.027+0000 7f093a0ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:40.073 INFO:teuthology.orchestra.run.smithi079.stderr:reweighted osd.3 to 1 (10000) 2024-04-24T01:58:40.078 INFO:tasks.rados.rados.0.smithi079.stdout:796: done (3 left) 2024-04-24T01:58:40.078 INFO:tasks.rados.rados.0.smithi079.stdout:797: read oid 42 snap -1 2024-04-24T01:58:40.078 INFO:tasks.rados.rados.0.smithi079.stdout:797: expect deleted 2024-04-24T01:58:40.078 INFO:tasks.rados.rados.0.smithi079.stdout:798: write oid 38 current snap is 93 2024-04-24T01:58:40.078 INFO:tasks.rados.rados.0.smithi079.stdout:794: finishing write tid 2 to smithi07929762-29 2024-04-24T01:58:40.079 INFO:tasks.rados.rados.0.smithi079.stdout:798: seq_num 390 ranges {554730=481113,1572095=737454} 2024-04-24T01:58:40.084 INFO:tasks.rados.rados.0.smithi079.stdout:798: writing smithi07929762-38 from 554730 to 1035843 tid 1 2024-04-24T01:58:40.088 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-04-24T01:58:40.088 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T01:58:40.095 INFO:tasks.rados.rados.0.smithi079.stdout:798: writing smithi07929762-38 from 1572095 to 2309549 tid 2 2024-04-24T01:58:40.095 INFO:tasks.rados.rados.0.smithi079.stdout:799: write (excl) oid 12 current snap is 93 2024-04-24T01:58:40.095 INFO:tasks.rados.rados.0.smithi079.stdout:791: finishing write tid 1 to smithi07929762-9 2024-04-24T01:58:40.095 INFO:tasks.rados.rados.0.smithi079.stdout:794: finishing write tid 3 to smithi07929762-29 2024-04-24T01:58:40.095 INFO:tasks.rados.rados.0.smithi079.stdout:794: finishing write tid 4 to smithi07929762-29 2024-04-24T01:58:40.095 INFO:tasks.rados.rados.0.smithi079.stdout:794: oid 29 updating version 0 to 189 2024-04-24T01:58:40.095 INFO:tasks.rados.rados.0.smithi079.stdout:794: oid 29 version 189 is already newer than 189 2024-04-24T01:58:40.095 INFO:tasks.rados.rados.0.smithi079.stdout:794: oid 29 version 189 is already newer than 189 2024-04-24T01:58:40.095 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 29 v 189 (ObjNum 388 snap 93 seq_num 388) dirty exists 2024-04-24T01:58:40.095 INFO:tasks.rados.rados.0.smithi079.stdout:794: left oid 29 (ObjNum 388 snap 93 seq_num 388) 2024-04-24T01:58:40.095 INFO:tasks.rados.rados.0.smithi079.stdout:791: finishing write tid 2 to smithi07929762-9 2024-04-24T01:58:40.096 INFO:tasks.rados.rados.0.smithi079.stdout:791: finishing write tid 3 to smithi07929762-9 2024-04-24T01:58:40.096 INFO:tasks.rados.rados.0.smithi079.stdout:791: finishing write tid 4 to smithi07929762-9 2024-04-24T01:58:40.096 INFO:tasks.rados.rados.0.smithi079.stdout:791: oid 9 updating version 0 to 278 2024-04-24T01:58:40.096 INFO:tasks.rados.rados.0.smithi079.stdout:791: oid 9 version 278 is already newer than 278 2024-04-24T01:58:40.096 INFO:tasks.rados.rados.0.smithi079.stdout:791: oid 9 version 278 is already newer than 278 2024-04-24T01:58:40.096 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 9 v 278 (ObjNum 387 snap 93 seq_num 387) dirty exists 2024-04-24T01:58:40.096 INFO:tasks.rados.rados.0.smithi079.stdout:791: left oid 9 (ObjNum 387 snap 93 seq_num 387) 2024-04-24T01:58:40.096 INFO:tasks.rados.rados.0.smithi079.stdout:792: expect (ObjNum 347 snap 82 seq_num 347) 2024-04-24T01:58:40.223 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:58:40.219+0000 7f1d65d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:40.227 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:58:40.227+0000 7f1d65d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:40.267 INFO:tasks.rados.rados.0.smithi079.stdout:799: seq_num 391 ranges {571537=556362,1630577=597465} 2024-04-24T01:58:40.274 INFO:tasks.rados.rados.0.smithi079.stdout:799: writing smithi07929762-12 from 571537 to 1127899 tid 1 2024-04-24T01:58:40.280 INFO:tasks.rados.rados.0.smithi079.stdout:799: writing smithi07929762-12 from 1630577 to 2228042 tid 2 2024-04-24T01:58:40.280 INFO:tasks.rados.rados.0.smithi079.stdout:791: done (5 left) 2024-04-24T01:58:40.280 INFO:tasks.rados.rados.0.smithi079.stdout:792: done (4 left) 2024-04-24T01:58:40.280 INFO:tasks.rados.rados.0.smithi079.stdout:794: done (3 left) 2024-04-24T01:58:40.280 INFO:tasks.rados.rados.0.smithi079.stdout:800: delete oid 44 current snap is 93 2024-04-24T01:58:40.280 INFO:tasks.rados.rados.0.smithi079.stdout:798: finishing write tid 1 to smithi07929762-38 2024-04-24T01:58:40.280 INFO:tasks.rados.rados.0.smithi079.stdout:798: finishing write tid 2 to smithi07929762-38 2024-04-24T01:58:40.281 INFO:tasks.rados.rados.0.smithi079.stdout:798: finishing write tid 3 to smithi07929762-38 2024-04-24T01:58:40.281 INFO:tasks.rados.rados.0.smithi079.stdout:798: finishing write tid 4 to smithi07929762-38 2024-04-24T01:58:40.281 INFO:tasks.rados.rados.0.smithi079.stdout:798: oid 38 updating version 0 to 213 2024-04-24T01:58:40.281 INFO:tasks.rados.rados.0.smithi079.stdout:798: oid 38 version 213 is already newer than 211 2024-04-24T01:58:40.281 INFO:tasks.rados.rados.0.smithi079.stdout:798: oid 38 version 213 is already newer than 213 2024-04-24T01:58:40.281 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 38 v 213 (ObjNum 390 snap 93 seq_num 390) dirty exists 2024-04-24T01:58:40.281 INFO:tasks.rados.rados.0.smithi079.stdout:798: left oid 38 (ObjNum 390 snap 93 seq_num 390) 2024-04-24T01:58:40.316 INFO:tasks.rados.rados.0.smithi079.stdout:799: finishing write tid 1 to smithi07929762-12 2024-04-24T01:58:40.332 INFO:tasks.rados.rados.0.smithi079.stdout:797: done (3 left) 2024-04-24T01:58:40.332 INFO:tasks.rados.rados.0.smithi079.stdout:798: done (2 left) 2024-04-24T01:58:40.332 INFO:tasks.rados.rados.0.smithi079.stdout:800: done (1 left) 2024-04-24T01:58:40.332 INFO:tasks.rados.rados.0.smithi079.stdout:801: read oid 26 snap -1 2024-04-24T01:58:40.332 INFO:tasks.rados.rados.0.smithi079.stdout:801: expect (ObjNum 366 snap 90 seq_num 366) 2024-04-24T01:58:40.332 INFO:tasks.rados.rados.0.smithi079.stdout:802: snap_remove snap 91 2024-04-24T01:58:40.428 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:58:40.428 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-24T01:58:40.441 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":298,"stamp":"2024-04-24T01:58:40.076409+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":88660665,"num_objects":42,"num_object_clones":5,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":1562,"num_read_kb":1017724,"num_write":1457,"num_write_kb":361866,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2979,"ondisk_log_size":2979,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":307920,"kb_used_data":179824,"kb_used_omap":0,"kb_used_meta":128064,"kb_avail":377179440,"statfs":{"total":386547056640,"available":386231746560,"internally_reserved":0,"allocated":184139776,"data_stored":183724552,"data_compressed":89889,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":131137536},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-505636,"num_objects":-9,"num_object_clones":-9,"num_object_copies":-27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":-1,"num_read":20,"num_read_kb":6608,"num_write":30,"num_write_kb":6528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.038468"},"pg_stats":[{"pgid":"3.f","version":"201'135","reported_seq":728,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-04-24T01:58:39.141002+0000","last_change":"2024-04-24T01:58:39.141002+0000","last_active":"2024-04-24T01:58:39.141002+0000","last_peered":"2024-04-24T01:58:39.141002+0000","last_clean":"2024-04-24T01:58:39.141002+0000","last_became_active":"2024-04-24T01:55:30.386726+0000","last_became_peered":"2024-04-24T01:55:30.386726+0000","last_unstale":"2024-04-24T01:58:39.141002+0000","last_undegraded":"2024-04-24T01:58:39.141002+0000","last_fullsized":"2024-04-24T01:58:39.141002+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":11,"snaptrim_duration":0,"stat_sum":{"num_bytes":2285818,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":100,"num_read_kb":89467,"num_write":58,"num_write_kb":16446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"201'240","reported_seq":800,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-04-24T01:58:39.088552+0000","last_change":"2024-04-24T01:58:39.094751+0000","last_active":"2024-04-24T01:58:39.088552+0000","last_peered":"2024-04-24T01:58:39.088552+0000","last_clean":"2024-04-24T01:58:39.088552+0000","last_became_active":"2024-04-24T01:55:30.321866+0000","last_became_peered":"2024-04-24T01:55:30.321866+0000","last_unstale":"2024-04-24T01:58:39.088552+0000","last_undegraded":"2024-04-24T01:58:39.088552+0000","last_fullsized":"2024-04-24T01:58:39.088552+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":240,"log_dups_size":0,"ondisk_log_size":240,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":17,"snaptrim_duration":0,"stat_sum":{"num_bytes":11328116,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":135,"num_read_kb":93306,"num_write":110,"num_write_kb":29873,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"182'185","reported_seq":739,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-04-24T01:58:38.079587+0000","last_change":"2024-04-24T01:58:37.214384+0000","last_active":"2024-04-24T01:58:38.079587+0000","last_peered":"2024-04-24T01:58:38.079587+0000","last_clean":"2024-04-24T01:58:38.079587+0000","last_became_active":"2024-04-24T01:55:30.320186+0000","last_became_peered":"2024-04-24T01:55:30.320186+0000","last_unstale":"2024-04-24T01:58:38.079587+0000","last_undegraded":"2024-04-24T01:58:38.079587+0000","last_fullsized":"2024-04-24T01:58:38.079587+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":185,"log_dups_size":0,"ondisk_log_size":185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":17,"snaptrim_duration":0,"stat_sum":{"num_bytes":6353112,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":76,"num_read_kb":46994,"num_write":95,"num_write_kb":25777,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"201'214","reported_seq":785,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-04-24T01:58:39.113036+0000","last_change":"2024-04-24T01:58:39.140524+0000","last_active":"2024-04-24T01:58:39.113036+0000","last_peered":"2024-04-24T01:58:39.113036+0000","last_clean":"2024-04-24T01:58:39.113036+0000","last_became_active":"2024-04-24T01:55:30.288081+0000","last_became_peered":"2024-04-24T01:55:30.288081+0000","last_unstale":"2024-04-24T01:58:39.113036+0000","last_undegraded":"2024-04-24T01:58:39.113036+0000","last_fullsized":"2024-04-24T01:58:39.113036+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":214,"log_dups_size":0,"ondisk_log_size":214,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":21,"snaptrim_duration":0,"stat_sum":{"num_bytes":5596830,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":117,"num_read_kb":84156,"num_write":89,"num_write_kb":25019,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"182'61","reported_seq":643,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-04-24T01:58:38.251861+0000","last_change":"2024-04-24T01:58:37.092662+0000","last_active":"2024-04-24T01:58:38.251861+0000","last_peered":"2024-04-24T01:58:38.251861+0000","last_clean":"2024-04-24T01:58:38.251861+0000","last_became_active":"2024-04-24T01:55:30.333815+0000","last_became_peered":"2024-04-24T01:55:30.333815+0000","last_unstale":"2024-04-24T01:58:38.251861+0000","last_undegraded":"2024-04-24T01:58:38.251861+0000","last_fullsized":"2024-04-24T01:58:38.251861+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":5,"snaptrim_duration":0,"stat_sum":{"num_bytes":3798382,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":32,"num_read_kb":17556,"num_write":36,"num_write_kb":10196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"201'415","reported_seq":939,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-04-24T01:58:39.120800+0000","last_change":"2024-04-24T01:58:39.134556+0000","last_active":"2024-04-24T01:58:39.120800+0000","last_peered":"2024-04-24T01:58:39.120800+0000","last_clean":"2024-04-24T01:58:39.120800+0000","last_became_active":"2024-04-24T01:55:30.315075+0000","last_became_peered":"2024-04-24T01:55:30.315075+0000","last_unstale":"2024-04-24T01:58:39.120800+0000","last_undegraded":"2024-04-24T01:58:39.120800+0000","last_fullsized":"2024-04-24T01:58:39.120800+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":415,"log_dups_size":0,"ondisk_log_size":415,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":37,"snaptrim_duration":0,"stat_sum":{"num_bytes":9882648,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":172,"num_read_kb":102271,"num_write":171,"num_write_kb":45947,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"183'66","reported_seq":654,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-04-24T01:58:39.141548+0000","last_change":"2024-04-24T01:58:39.149264+0000","last_active":"2024-04-24T01:58:39.141548+0000","last_peered":"2024-04-24T01:58:39.141548+0000","last_clean":"2024-04-24T01:58:39.141548+0000","last_became_active":"2024-04-24T01:55:30.350300+0000","last_became_peered":"2024-04-24T01:55:30.350300+0000","last_unstale":"2024-04-24T01:58:39.141548+0000","last_undegraded":"2024-04-24T01:58:39.141548+0000","last_fullsized":"2024-04-24T01:58:39.141548+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0,"stat_sum":{"num_bytes":3596731,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":33,"num_read_kb":16964,"num_write":41,"num_write_kb":10355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"163'8","reported_seq":581,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-04-24T01:58:38.089801+0000","last_change":"2024-04-24T01:58:37.226831+0000","last_active":"2024-04-24T01:58:38.089801+0000","last_peered":"2024-04-24T01:58:38.089801+0000","last_clean":"2024-04-24T01:58:38.089801+0000","last_became_active":"2024-04-24T01:55:30.376813+0000","last_became_peered":"2024-04-24T01:55:30.376813+0000","last_unstale":"2024-04-24T01:58:38.089801+0000","last_undegraded":"2024-04-24T01:58:38.089801+0000","last_fullsized":"2024-04-24T01:58:38.089801+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'97","reported_seq":636,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-04-24T01:58:38.240462+0000","last_change":"2024-04-24T01:55:16.196470+0000","last_active":"2024-04-24T01:58:38.240462+0000","last_peered":"2024-04-24T01:58:38.240462+0000","last_clean":"2024-04-24T01:58:38.240462+0000","last_became_active":"2024-04-24T01:55:16.190464+0000","last_became_peered":"2024-04-24T01:55:16.190464+0000","last_unstale":"2024-04-24T01:58:38.240462+0000","last_undegraded":"2024-04-24T01:58:38.240462+0000","last_fullsized":"2024-04-24T01:58:38.240462+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:15.127115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:15.127115+0000","last_clean_scrub_stamp":"2024-04-24T01:55:15.127115+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":495,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-04-24T01:58:38.087296+0000","last_change":"2024-04-24T01:55:18.236148+0000","last_active":"2024-04-24T01:58:38.087296+0000","last_peered":"2024-04-24T01:58:38.087296+0000","last_clean":"2024-04-24T01:58:38.087296+0000","last_became_active":"2024-04-24T01:55:18.230154+0000","last_became_peered":"2024-04-24T01:55:18.230154+0000","last_unstale":"2024-04-24T01:58:38.087296+0000","last_undegraded":"2024-04-24T01:58:38.087296+0000","last_fullsized":"2024-04-24T01:58:38.087296+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"201'314","reported_seq":874,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-04-24T01:58:38.087031+0000","last_change":"2024-04-24T01:58:37.316299+0000","last_active":"2024-04-24T01:58:38.087031+0000","last_peered":"2024-04-24T01:58:38.087031+0000","last_clean":"2024-04-24T01:58:38.087031+0000","last_became_active":"2024-04-24T01:55:30.316222+0000","last_became_peered":"2024-04-24T01:55:30.316222+0000","last_unstale":"2024-04-24T01:58:38.087031+0000","last_undegraded":"2024-04-24T01:58:38.087031+0000","last_fullsized":"2024-04-24T01:58:38.087031+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":314,"log_dups_size":0,"ondisk_log_size":314,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":24,"snaptrim_duration":0,"stat_sum":{"num_bytes":10739904,"num_objects":6,"num_object_clones":2,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":162,"num_read_kb":107369,"num_write":158,"num_write_kb":44575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":508,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-04-24T01:58:39.138421+0000","last_change":"2024-04-24T01:55:18.265905+0000","last_active":"2024-04-24T01:58:39.138421+0000","last_peered":"2024-04-24T01:58:39.138421+0000","last_clean":"2024-04-24T01:58:39.138421+0000","last_became_active":"2024-04-24T01:55:18.257020+0000","last_became_peered":"2024-04-24T01:55:18.257020+0000","last_unstale":"2024-04-24T01:58:39.138421+0000","last_undegraded":"2024-04-24T01:58:39.138421+0000","last_fullsized":"2024-04-24T01:58:39.138421+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"201'264","reported_seq":807,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-04-24T01:58:38.241863+0000","last_change":"2024-04-24T01:58:37.198398+0000","last_active":"2024-04-24T01:58:38.241863+0000","last_peered":"2024-04-24T01:58:38.241863+0000","last_clean":"2024-04-24T01:58:38.241863+0000","last_became_active":"2024-04-24T01:55:30.344129+0000","last_became_peered":"2024-04-24T01:55:30.344129+0000","last_unstale":"2024-04-24T01:58:38.241863+0000","last_undegraded":"2024-04-24T01:58:38.241863+0000","last_fullsized":"2024-04-24T01:58:38.241863+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":264,"log_dups_size":0,"ondisk_log_size":264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":18,"snaptrim_duration":0,"stat_sum":{"num_bytes":6420472,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":102,"num_read_kb":60236,"num_write":114,"num_write_kb":32496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":495,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-04-24T01:58:38.078901+0000","last_change":"2024-04-24T01:55:18.215251+0000","last_active":"2024-04-24T01:58:38.078901+0000","last_peered":"2024-04-24T01:58:38.078901+0000","last_clean":"2024-04-24T01:58:38.078901+0000","last_became_active":"2024-04-24T01:55:18.206605+0000","last_became_peered":"2024-04-24T01:55:18.206605+0000","last_unstale":"2024-04-24T01:58:38.078901+0000","last_undegraded":"2024-04-24T01:58:38.078901+0000","last_fullsized":"2024-04-24T01:58:38.078901+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"201'131","reported_seq":706,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-04-24T01:58:38.253362+0000","last_change":"2024-04-24T01:58:37.129566+0000","last_active":"2024-04-24T01:58:38.253362+0000","last_peered":"2024-04-24T01:58:38.253362+0000","last_clean":"2024-04-24T01:58:38.253362+0000","last_became_active":"2024-04-24T01:55:30.401366+0000","last_became_peered":"2024-04-24T01:55:30.401366+0000","last_unstale":"2024-04-24T01:58:38.253362+0000","last_undegraded":"2024-04-24T01:58:38.253362+0000","last_fullsized":"2024-04-24T01:58:38.253362+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":131,"log_dups_size":0,"ondisk_log_size":131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":10,"snaptrim_duration":0,"stat_sum":{"num_bytes":2218590,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":82,"num_read_kb":65583,"num_write":42,"num_write_kb":10680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":508,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-04-24T01:58:39.132749+0000","last_change":"2024-04-24T01:55:18.230893+0000","last_active":"2024-04-24T01:58:39.132749+0000","last_peered":"2024-04-24T01:58:39.132749+0000","last_clean":"2024-04-24T01:58:39.132749+0000","last_became_active":"2024-04-24T01:55:18.225188+0000","last_became_peered":"2024-04-24T01:55:18.225188+0000","last_unstale":"2024-04-24T01:58:39.132749+0000","last_undegraded":"2024-04-24T01:58:39.132749+0000","last_fullsized":"2024-04-24T01:58:39.132749+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"201'216","reported_seq":762,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-04-24T01:58:39.086597+0000","last_change":"2024-04-24T01:58:39.095014+0000","last_active":"2024-04-24T01:58:39.086597+0000","last_peered":"2024-04-24T01:58:39.086597+0000","last_clean":"2024-04-24T01:58:39.086597+0000","last_became_active":"2024-04-24T01:55:30.343318+0000","last_became_peered":"2024-04-24T01:55:30.343318+0000","last_unstale":"2024-04-24T01:58:39.086597+0000","last_undegraded":"2024-04-24T01:58:39.086597+0000","last_fullsized":"2024-04-24T01:58:39.086597+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":216,"log_dups_size":0,"ondisk_log_size":216,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":16,"snaptrim_duration":0,"stat_sum":{"num_bytes":4571636,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":98,"num_read_kb":79025,"num_write":96,"num_write_kb":26701,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"201'274","reported_seq":829,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-04-24T01:58:38.089049+0000","last_change":"2024-04-24T01:58:37.264791+0000","last_active":"2024-04-24T01:58:38.089049+0000","last_peered":"2024-04-24T01:58:38.089049+0000","last_clean":"2024-04-24T01:58:38.089049+0000","last_became_active":"2024-04-24T01:55:30.345308+0000","last_became_peered":"2024-04-24T01:55:30.345308+0000","last_unstale":"2024-04-24T01:58:38.089049+0000","last_undegraded":"2024-04-24T01:58:38.089049+0000","last_fullsized":"2024-04-24T01:58:38.089049+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":274,"log_dups_size":0,"ondisk_log_size":274,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":21,"snaptrim_duration":0,"stat_sum":{"num_bytes":11109625,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":149,"num_read_kb":102577,"num_write":116,"num_write_kb":29821,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":490,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-04-24T01:58:39.096655+0000","last_change":"2024-04-24T01:55:18.224238+0000","last_active":"2024-04-24T01:58:39.096655+0000","last_peered":"2024-04-24T01:58:39.096655+0000","last_clean":"2024-04-24T01:58:39.096655+0000","last_became_active":"2024-04-24T01:55:18.219131+0000","last_became_peered":"2024-04-24T01:55:18.219131+0000","last_unstale":"2024-04-24T01:58:39.096655+0000","last_undegraded":"2024-04-24T01:58:39.096655+0000","last_fullsized":"2024-04-24T01:58:39.096655+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"174'54","reported_seq":625,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-04-24T01:58:39.095792+0000","last_change":"2024-04-24T01:58:39.103557+0000","last_active":"2024-04-24T01:58:39.095792+0000","last_peered":"2024-04-24T01:58:39.095792+0000","last_clean":"2024-04-24T01:58:39.095792+0000","last_became_active":"2024-04-24T01:55:30.319958+0000","last_became_peered":"2024-04-24T01:55:30.319958+0000","last_unstale":"2024-04-24T01:58:39.095792+0000","last_undegraded":"2024-04-24T01:58:39.095792+0000","last_fullsized":"2024-04-24T01:58:39.095792+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":5,"snaptrim_duration":0,"stat_sum":{"num_bytes":2823609,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":26,"num_read_kb":19608,"num_write":37,"num_write_kb":10363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":503,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-04-24T01:58:38.249232+0000","last_change":"2024-04-24T01:55:18.235763+0000","last_active":"2024-04-24T01:58:38.249232+0000","last_peered":"2024-04-24T01:58:38.249232+0000","last_clean":"2024-04-24T01:58:38.249232+0000","last_became_active":"2024-04-24T01:55:18.229175+0000","last_became_peered":"2024-04-24T01:55:18.229175+0000","last_unstale":"2024-04-24T01:58:38.249232+0000","last_undegraded":"2024-04-24T01:58:38.249232+0000","last_fullsized":"2024-04-24T01:58:38.249232+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"201'95","reported_seq":667,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-04-24T01:58:38.243583+0000","last_change":"2024-04-24T01:58:37.187841+0000","last_active":"2024-04-24T01:58:38.243583+0000","last_peered":"2024-04-24T01:58:38.243583+0000","last_clean":"2024-04-24T01:58:38.243583+0000","last_became_active":"2024-04-24T01:55:30.372690+0000","last_became_peered":"2024-04-24T01:55:30.372690+0000","last_unstale":"2024-04-24T01:58:38.243583+0000","last_undegraded":"2024-04-24T01:58:38.243583+0000","last_fullsized":"2024-04-24T01:58:38.243583+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":7,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":46,"num_read_kb":30459,"num_write":40,"num_write_kb":11003,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":490,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-04-24T01:58:39.097577+0000","last_change":"2024-04-24T01:55:18.278351+0000","last_active":"2024-04-24T01:58:39.097577+0000","last_peered":"2024-04-24T01:58:39.097577+0000","last_clean":"2024-04-24T01:58:39.097577+0000","last_became_active":"2024-04-24T01:55:18.273668+0000","last_became_peered":"2024-04-24T01:55:18.273668+0000","last_unstale":"2024-04-24T01:58:39.097577+0000","last_undegraded":"2024-04-24T01:58:39.097577+0000","last_fullsized":"2024-04-24T01:58:39.097577+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"201'210","reported_seq":781,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-04-24T01:58:39.099609+0000","last_change":"2024-04-24T01:58:39.104137+0000","last_active":"2024-04-24T01:58:39.099609+0000","last_peered":"2024-04-24T01:58:39.099609+0000","last_clean":"2024-04-24T01:58:39.099609+0000","last_became_active":"2024-04-24T01:55:30.363123+0000","last_became_peered":"2024-04-24T01:55:30.363123+0000","last_unstale":"2024-04-24T01:58:39.099609+0000","last_undegraded":"2024-04-24T01:58:39.099609+0000","last_fullsized":"2024-04-24T01:58:39.099609+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":210,"log_dups_size":0,"ondisk_log_size":210,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":20,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344824,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":127,"num_read_kb":102061,"num_write":112,"num_write_kb":29872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":495,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-04-24T01:58:38.090910+0000","last_change":"2024-04-24T01:55:18.241117+0000","last_active":"2024-04-24T01:58:38.090910+0000","last_peered":"2024-04-24T01:58:38.090910+0000","last_clean":"2024-04-24T01:58:38.090910+0000","last_became_active":"2024-04-24T01:55:18.234313+0000","last_became_peered":"2024-04-24T01:55:18.234313+0000","last_unstale":"2024-04-24T01:58:38.090910+0000","last_undegraded":"2024-04-24T01:58:38.090910+0000","last_fullsized":"2024-04-24T01:58:38.090910+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":88070297,"num_objects":40,"num_object_clones":5,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":1459,"num_read_kb":1017633,"num_write":1320,"num_write_kb":359972,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":179335168,"data_stored":183890608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2882,"ondisk_log_size":2882,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":1}],"osd_stats":[{"osd":3,"up_from":10,"seq":42949673164,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":86580,"kb_used_data":51436,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94285260,"statfs":{"total":96636764160,"available":96548106240,"internally_reserved":0,"allocated":52670464,"data_stored":52911661,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673164,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":70436,"kb_used_data":39452,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94301404,"statfs":{"total":96636764160,"available":96564637696,"internally_reserved":0,"allocated":40398848,"data_stored":39592160,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705869,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":73092,"kb_used_data":42108,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94298748,"statfs":{"total":96636764160,"available":96561917952,"internally_reserved":0,"allocated":43118592,"data_stored":43478779,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705868,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":77812,"kb_used_data":46828,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94294028,"statfs":{"total":96636764160,"available":96557084672,"internally_reserved":0,"allocated":47951872,"data_stored":47741952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":42131456,"data_stored":43706562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":46301184,"data_stored":47026943,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":39219200,"data_stored":40017659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":51683328,"data_stored":53139444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T01:58:40.443 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T01:58:40.548 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:58:40.544+0000 7f642266d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:40.552 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:58:40.551+0000 7f642266d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:40.735 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:58:40.736 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-24T01:58:40.749 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":298,"stamp":"2024-04-24T01:58:40.076409+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":88660665,"num_objects":42,"num_object_clones":5,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":1562,"num_read_kb":1017724,"num_write":1457,"num_write_kb":361866,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2979,"ondisk_log_size":2979,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":307920,"kb_used_data":179824,"kb_used_omap":0,"kb_used_meta":128064,"kb_avail":377179440,"statfs":{"total":386547056640,"available":386231746560,"internally_reserved":0,"allocated":184139776,"data_stored":183724552,"data_compressed":89889,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":131137536},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-505636,"num_objects":-9,"num_object_clones":-9,"num_object_copies":-27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":-1,"num_read":20,"num_read_kb":6608,"num_write":30,"num_write_kb":6528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.038468"},"pg_stats":[{"pgid":"3.f","version":"201'135","reported_seq":728,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-04-24T01:58:39.141002+0000","last_change":"2024-04-24T01:58:39.141002+0000","last_active":"2024-04-24T01:58:39.141002+0000","last_peered":"2024-04-24T01:58:39.141002+0000","last_clean":"2024-04-24T01:58:39.141002+0000","last_became_active":"2024-04-24T01:55:30.386726+0000","last_became_peered":"2024-04-24T01:55:30.386726+0000","last_unstale":"2024-04-24T01:58:39.141002+0000","last_undegraded":"2024-04-24T01:58:39.141002+0000","last_fullsized":"2024-04-24T01:58:39.141002+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":11,"snaptrim_duration":0,"stat_sum":{"num_bytes":2285818,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":100,"num_read_kb":89467,"num_write":58,"num_write_kb":16446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"201'240","reported_seq":800,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-04-24T01:58:39.088552+0000","last_change":"2024-04-24T01:58:39.094751+0000","last_active":"2024-04-24T01:58:39.088552+0000","last_peered":"2024-04-24T01:58:39.088552+0000","last_clean":"2024-04-24T01:58:39.088552+0000","last_became_active":"2024-04-24T01:55:30.321866+0000","last_became_peered":"2024-04-24T01:55:30.321866+0000","last_unstale":"2024-04-24T01:58:39.088552+0000","last_undegraded":"2024-04-24T01:58:39.088552+0000","last_fullsized":"2024-04-24T01:58:39.088552+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":240,"log_dups_size":0,"ondisk_log_size":240,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":17,"snaptrim_duration":0,"stat_sum":{"num_bytes":11328116,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":135,"num_read_kb":93306,"num_write":110,"num_write_kb":29873,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"182'185","reported_seq":739,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-04-24T01:58:38.079587+0000","last_change":"2024-04-24T01:58:37.214384+0000","last_active":"2024-04-24T01:58:38.079587+0000","last_peered":"2024-04-24T01:58:38.079587+0000","last_clean":"2024-04-24T01:58:38.079587+0000","last_became_active":"2024-04-24T01:55:30.320186+0000","last_became_peered":"2024-04-24T01:55:30.320186+0000","last_unstale":"2024-04-24T01:58:38.079587+0000","last_undegraded":"2024-04-24T01:58:38.079587+0000","last_fullsized":"2024-04-24T01:58:38.079587+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":185,"log_dups_size":0,"ondisk_log_size":185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":17,"snaptrim_duration":0,"stat_sum":{"num_bytes":6353112,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":76,"num_read_kb":46994,"num_write":95,"num_write_kb":25777,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"201'214","reported_seq":785,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-04-24T01:58:39.113036+0000","last_change":"2024-04-24T01:58:39.140524+0000","last_active":"2024-04-24T01:58:39.113036+0000","last_peered":"2024-04-24T01:58:39.113036+0000","last_clean":"2024-04-24T01:58:39.113036+0000","last_became_active":"2024-04-24T01:55:30.288081+0000","last_became_peered":"2024-04-24T01:55:30.288081+0000","last_unstale":"2024-04-24T01:58:39.113036+0000","last_undegraded":"2024-04-24T01:58:39.113036+0000","last_fullsized":"2024-04-24T01:58:39.113036+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":214,"log_dups_size":0,"ondisk_log_size":214,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":21,"snaptrim_duration":0,"stat_sum":{"num_bytes":5596830,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":117,"num_read_kb":84156,"num_write":89,"num_write_kb":25019,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"182'61","reported_seq":643,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-04-24T01:58:38.251861+0000","last_change":"2024-04-24T01:58:37.092662+0000","last_active":"2024-04-24T01:58:38.251861+0000","last_peered":"2024-04-24T01:58:38.251861+0000","last_clean":"2024-04-24T01:58:38.251861+0000","last_became_active":"2024-04-24T01:55:30.333815+0000","last_became_peered":"2024-04-24T01:55:30.333815+0000","last_unstale":"2024-04-24T01:58:38.251861+0000","last_undegraded":"2024-04-24T01:58:38.251861+0000","last_fullsized":"2024-04-24T01:58:38.251861+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":5,"snaptrim_duration":0,"stat_sum":{"num_bytes":3798382,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":32,"num_read_kb":17556,"num_write":36,"num_write_kb":10196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"201'415","reported_seq":939,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-04-24T01:58:39.120800+0000","last_change":"2024-04-24T01:58:39.134556+0000","last_active":"2024-04-24T01:58:39.120800+0000","last_peered":"2024-04-24T01:58:39.120800+0000","last_clean":"2024-04-24T01:58:39.120800+0000","last_became_active":"2024-04-24T01:55:30.315075+0000","last_became_peered":"2024-04-24T01:55:30.315075+0000","last_unstale":"2024-04-24T01:58:39.120800+0000","last_undegraded":"2024-04-24T01:58:39.120800+0000","last_fullsized":"2024-04-24T01:58:39.120800+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":415,"log_dups_size":0,"ondisk_log_size":415,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":37,"snaptrim_duration":0,"stat_sum":{"num_bytes":9882648,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":172,"num_read_kb":102271,"num_write":171,"num_write_kb":45947,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"183'66","reported_seq":654,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-04-24T01:58:39.141548+0000","last_change":"2024-04-24T01:58:39.149264+0000","last_active":"2024-04-24T01:58:39.141548+0000","last_peered":"2024-04-24T01:58:39.141548+0000","last_clean":"2024-04-24T01:58:39.141548+0000","last_became_active":"2024-04-24T01:55:30.350300+0000","last_became_peered":"2024-04-24T01:55:30.350300+0000","last_unstale":"2024-04-24T01:58:39.141548+0000","last_undegraded":"2024-04-24T01:58:39.141548+0000","last_fullsized":"2024-04-24T01:58:39.141548+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0,"stat_sum":{"num_bytes":3596731,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":33,"num_read_kb":16964,"num_write":41,"num_write_kb":10355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"163'8","reported_seq":581,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-04-24T01:58:38.089801+0000","last_change":"2024-04-24T01:58:37.226831+0000","last_active":"2024-04-24T01:58:38.089801+0000","last_peered":"2024-04-24T01:58:38.089801+0000","last_clean":"2024-04-24T01:58:38.089801+0000","last_became_active":"2024-04-24T01:55:30.376813+0000","last_became_peered":"2024-04-24T01:55:30.376813+0000","last_unstale":"2024-04-24T01:58:38.089801+0000","last_undegraded":"2024-04-24T01:58:38.089801+0000","last_fullsized":"2024-04-24T01:58:38.089801+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'97","reported_seq":636,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-04-24T01:58:38.240462+0000","last_change":"2024-04-24T01:55:16.196470+0000","last_active":"2024-04-24T01:58:38.240462+0000","last_peered":"2024-04-24T01:58:38.240462+0000","last_clean":"2024-04-24T01:58:38.240462+0000","last_became_active":"2024-04-24T01:55:16.190464+0000","last_became_peered":"2024-04-24T01:55:16.190464+0000","last_unstale":"2024-04-24T01:58:38.240462+0000","last_undegraded":"2024-04-24T01:58:38.240462+0000","last_fullsized":"2024-04-24T01:58:38.240462+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:15.127115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:15.127115+0000","last_clean_scrub_stamp":"2024-04-24T01:55:15.127115+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":495,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-04-24T01:58:38.087296+0000","last_change":"2024-04-24T01:55:18.236148+0000","last_active":"2024-04-24T01:58:38.087296+0000","last_peered":"2024-04-24T01:58:38.087296+0000","last_clean":"2024-04-24T01:58:38.087296+0000","last_became_active":"2024-04-24T01:55:18.230154+0000","last_became_peered":"2024-04-24T01:55:18.230154+0000","last_unstale":"2024-04-24T01:58:38.087296+0000","last_undegraded":"2024-04-24T01:58:38.087296+0000","last_fullsized":"2024-04-24T01:58:38.087296+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"201'314","reported_seq":874,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-04-24T01:58:38.087031+0000","last_change":"2024-04-24T01:58:37.316299+0000","last_active":"2024-04-24T01:58:38.087031+0000","last_peered":"2024-04-24T01:58:38.087031+0000","last_clean":"2024-04-24T01:58:38.087031+0000","last_became_active":"2024-04-24T01:55:30.316222+0000","last_became_peered":"2024-04-24T01:55:30.316222+0000","last_unstale":"2024-04-24T01:58:38.087031+0000","last_undegraded":"2024-04-24T01:58:38.087031+0000","last_fullsized":"2024-04-24T01:58:38.087031+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":314,"log_dups_size":0,"ondisk_log_size":314,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":24,"snaptrim_duration":0,"stat_sum":{"num_bytes":10739904,"num_objects":6,"num_object_clones":2,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":162,"num_read_kb":107369,"num_write":158,"num_write_kb":44575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":508,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-04-24T01:58:39.138421+0000","last_change":"2024-04-24T01:55:18.265905+0000","last_active":"2024-04-24T01:58:39.138421+0000","last_peered":"2024-04-24T01:58:39.138421+0000","last_clean":"2024-04-24T01:58:39.138421+0000","last_became_active":"2024-04-24T01:55:18.257020+0000","last_became_peered":"2024-04-24T01:55:18.257020+0000","last_unstale":"2024-04-24T01:58:39.138421+0000","last_undegraded":"2024-04-24T01:58:39.138421+0000","last_fullsized":"2024-04-24T01:58:39.138421+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"201'264","reported_seq":807,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-04-24T01:58:38.241863+0000","last_change":"2024-04-24T01:58:37.198398+0000","last_active":"2024-04-24T01:58:38.241863+0000","last_peered":"2024-04-24T01:58:38.241863+0000","last_clean":"2024-04-24T01:58:38.241863+0000","last_became_active":"2024-04-24T01:55:30.344129+0000","last_became_peered":"2024-04-24T01:55:30.344129+0000","last_unstale":"2024-04-24T01:58:38.241863+0000","last_undegraded":"2024-04-24T01:58:38.241863+0000","last_fullsized":"2024-04-24T01:58:38.241863+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":264,"log_dups_size":0,"ondisk_log_size":264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":18,"snaptrim_duration":0,"stat_sum":{"num_bytes":6420472,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":102,"num_read_kb":60236,"num_write":114,"num_write_kb":32496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":495,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-04-24T01:58:38.078901+0000","last_change":"2024-04-24T01:55:18.215251+0000","last_active":"2024-04-24T01:58:38.078901+0000","last_peered":"2024-04-24T01:58:38.078901+0000","last_clean":"2024-04-24T01:58:38.078901+0000","last_became_active":"2024-04-24T01:55:18.206605+0000","last_became_peered":"2024-04-24T01:55:18.206605+0000","last_unstale":"2024-04-24T01:58:38.078901+0000","last_undegraded":"2024-04-24T01:58:38.078901+0000","last_fullsized":"2024-04-24T01:58:38.078901+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"201'131","reported_seq":706,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-04-24T01:58:38.253362+0000","last_change":"2024-04-24T01:58:37.129566+0000","last_active":"2024-04-24T01:58:38.253362+0000","last_peered":"2024-04-24T01:58:38.253362+0000","last_clean":"2024-04-24T01:58:38.253362+0000","last_became_active":"2024-04-24T01:55:30.401366+0000","last_became_peered":"2024-04-24T01:55:30.401366+0000","last_unstale":"2024-04-24T01:58:38.253362+0000","last_undegraded":"2024-04-24T01:58:38.253362+0000","last_fullsized":"2024-04-24T01:58:38.253362+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":131,"log_dups_size":0,"ondisk_log_size":131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":10,"snaptrim_duration":0,"stat_sum":{"num_bytes":2218590,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":82,"num_read_kb":65583,"num_write":42,"num_write_kb":10680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":508,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-04-24T01:58:39.132749+0000","last_change":"2024-04-24T01:55:18.230893+0000","last_active":"2024-04-24T01:58:39.132749+0000","last_peered":"2024-04-24T01:58:39.132749+0000","last_clean":"2024-04-24T01:58:39.132749+0000","last_became_active":"2024-04-24T01:55:18.225188+0000","last_became_peered":"2024-04-24T01:55:18.225188+0000","last_unstale":"2024-04-24T01:58:39.132749+0000","last_undegraded":"2024-04-24T01:58:39.132749+0000","last_fullsized":"2024-04-24T01:58:39.132749+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"201'216","reported_seq":762,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-04-24T01:58:39.086597+0000","last_change":"2024-04-24T01:58:39.095014+0000","last_active":"2024-04-24T01:58:39.086597+0000","last_peered":"2024-04-24T01:58:39.086597+0000","last_clean":"2024-04-24T01:58:39.086597+0000","last_became_active":"2024-04-24T01:55:30.343318+0000","last_became_peered":"2024-04-24T01:55:30.343318+0000","last_unstale":"2024-04-24T01:58:39.086597+0000","last_undegraded":"2024-04-24T01:58:39.086597+0000","last_fullsized":"2024-04-24T01:58:39.086597+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":216,"log_dups_size":0,"ondisk_log_size":216,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":16,"snaptrim_duration":0,"stat_sum":{"num_bytes":4571636,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":98,"num_read_kb":79025,"num_write":96,"num_write_kb":26701,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"201'274","reported_seq":829,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-04-24T01:58:38.089049+0000","last_change":"2024-04-24T01:58:37.264791+0000","last_active":"2024-04-24T01:58:38.089049+0000","last_peered":"2024-04-24T01:58:38.089049+0000","last_clean":"2024-04-24T01:58:38.089049+0000","last_became_active":"2024-04-24T01:55:30.345308+0000","last_became_peered":"2024-04-24T01:55:30.345308+0000","last_unstale":"2024-04-24T01:58:38.089049+0000","last_undegraded":"2024-04-24T01:58:38.089049+0000","last_fullsized":"2024-04-24T01:58:38.089049+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":274,"log_dups_size":0,"ondisk_log_size":274,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":21,"snaptrim_duration":0,"stat_sum":{"num_bytes":11109625,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":149,"num_read_kb":102577,"num_write":116,"num_write_kb":29821,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":490,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-04-24T01:58:39.096655+0000","last_change":"2024-04-24T01:55:18.224238+0000","last_active":"2024-04-24T01:58:39.096655+0000","last_peered":"2024-04-24T01:58:39.096655+0000","last_clean":"2024-04-24T01:58:39.096655+0000","last_became_active":"2024-04-24T01:55:18.219131+0000","last_became_peered":"2024-04-24T01:55:18.219131+0000","last_unstale":"2024-04-24T01:58:39.096655+0000","last_undegraded":"2024-04-24T01:58:39.096655+0000","last_fullsized":"2024-04-24T01:58:39.096655+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"174'54","reported_seq":625,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-04-24T01:58:39.095792+0000","last_change":"2024-04-24T01:58:39.103557+0000","last_active":"2024-04-24T01:58:39.095792+0000","last_peered":"2024-04-24T01:58:39.095792+0000","last_clean":"2024-04-24T01:58:39.095792+0000","last_became_active":"2024-04-24T01:55:30.319958+0000","last_became_peered":"2024-04-24T01:55:30.319958+0000","last_unstale":"2024-04-24T01:58:39.095792+0000","last_undegraded":"2024-04-24T01:58:39.095792+0000","last_fullsized":"2024-04-24T01:58:39.095792+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":5,"snaptrim_duration":0,"stat_sum":{"num_bytes":2823609,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":26,"num_read_kb":19608,"num_write":37,"num_write_kb":10363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":503,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-04-24T01:58:38.249232+0000","last_change":"2024-04-24T01:55:18.235763+0000","last_active":"2024-04-24T01:58:38.249232+0000","last_peered":"2024-04-24T01:58:38.249232+0000","last_clean":"2024-04-24T01:58:38.249232+0000","last_became_active":"2024-04-24T01:55:18.229175+0000","last_became_peered":"2024-04-24T01:55:18.229175+0000","last_unstale":"2024-04-24T01:58:38.249232+0000","last_undegraded":"2024-04-24T01:58:38.249232+0000","last_fullsized":"2024-04-24T01:58:38.249232+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"201'95","reported_seq":667,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-04-24T01:58:38.243583+0000","last_change":"2024-04-24T01:58:37.187841+0000","last_active":"2024-04-24T01:58:38.243583+0000","last_peered":"2024-04-24T01:58:38.243583+0000","last_clean":"2024-04-24T01:58:38.243583+0000","last_became_active":"2024-04-24T01:55:30.372690+0000","last_became_peered":"2024-04-24T01:55:30.372690+0000","last_unstale":"2024-04-24T01:58:38.243583+0000","last_undegraded":"2024-04-24T01:58:38.243583+0000","last_fullsized":"2024-04-24T01:58:38.243583+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":7,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":46,"num_read_kb":30459,"num_write":40,"num_write_kb":11003,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":490,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-04-24T01:58:39.097577+0000","last_change":"2024-04-24T01:55:18.278351+0000","last_active":"2024-04-24T01:58:39.097577+0000","last_peered":"2024-04-24T01:58:39.097577+0000","last_clean":"2024-04-24T01:58:39.097577+0000","last_became_active":"2024-04-24T01:55:18.273668+0000","last_became_peered":"2024-04-24T01:55:18.273668+0000","last_unstale":"2024-04-24T01:58:39.097577+0000","last_undegraded":"2024-04-24T01:58:39.097577+0000","last_fullsized":"2024-04-24T01:58:39.097577+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"201'210","reported_seq":781,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-04-24T01:58:39.099609+0000","last_change":"2024-04-24T01:58:39.104137+0000","last_active":"2024-04-24T01:58:39.099609+0000","last_peered":"2024-04-24T01:58:39.099609+0000","last_clean":"2024-04-24T01:58:39.099609+0000","last_became_active":"2024-04-24T01:55:30.363123+0000","last_became_peered":"2024-04-24T01:55:30.363123+0000","last_unstale":"2024-04-24T01:58:39.099609+0000","last_undegraded":"2024-04-24T01:58:39.099609+0000","last_fullsized":"2024-04-24T01:58:39.099609+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":210,"log_dups_size":0,"ondisk_log_size":210,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":20,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344824,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":127,"num_read_kb":102061,"num_write":112,"num_write_kb":29872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":495,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-04-24T01:58:38.090910+0000","last_change":"2024-04-24T01:55:18.241117+0000","last_active":"2024-04-24T01:58:38.090910+0000","last_peered":"2024-04-24T01:58:38.090910+0000","last_clean":"2024-04-24T01:58:38.090910+0000","last_became_active":"2024-04-24T01:55:18.234313+0000","last_became_peered":"2024-04-24T01:55:18.234313+0000","last_unstale":"2024-04-24T01:58:38.090910+0000","last_undegraded":"2024-04-24T01:58:38.090910+0000","last_fullsized":"2024-04-24T01:58:38.090910+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":88070297,"num_objects":40,"num_object_clones":5,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":1459,"num_read_kb":1017633,"num_write":1320,"num_write_kb":359972,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":179335168,"data_stored":183890608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2882,"ondisk_log_size":2882,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":1}],"osd_stats":[{"osd":3,"up_from":10,"seq":42949673164,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":86580,"kb_used_data":51436,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94285260,"statfs":{"total":96636764160,"available":96548106240,"internally_reserved":0,"allocated":52670464,"data_stored":52911661,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673164,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":70436,"kb_used_data":39452,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94301404,"statfs":{"total":96636764160,"available":96564637696,"internally_reserved":0,"allocated":40398848,"data_stored":39592160,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705869,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":73092,"kb_used_data":42108,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94298748,"statfs":{"total":96636764160,"available":96561917952,"internally_reserved":0,"allocated":43118592,"data_stored":43478779,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705868,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":77812,"kb_used_data":46828,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94294028,"statfs":{"total":96636764160,"available":96557084672,"internally_reserved":0,"allocated":47951872,"data_stored":47741952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":42131456,"data_stored":43706562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":46301184,"data_stored":47026943,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":39219200,"data_stored":40017659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":51683328,"data_stored":53139444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T01:58:40.750 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-04-24T01:58:40.750 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-04-24T01:58:40.751 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T01:58:40.856 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:58:40.852+0000 7f31a4c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:40.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:58:40.859+0000 7f31a4c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:41.046 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:58:41.046 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-24T01:58:41.059 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":299,"stamp":"2024-04-24T01:58:40.980490+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":88660665,"num_objects":42,"num_object_clones":5,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":1562,"num_read_kb":1017724,"num_write":1457,"num_write_kb":361866,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2979,"ondisk_log_size":2979,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":307920,"kb_used_data":179824,"kb_used_omap":0,"kb_used_meta":128064,"kb_avail":377179440,"statfs":{"total":386547056640,"available":386231746560,"internally_reserved":0,"allocated":184139776,"data_stored":183724552,"data_compressed":89889,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":131137536},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-505636,"num_objects":-9,"num_object_clones":-9,"num_object_copies":-27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":-1,"num_read":20,"num_read_kb":6608,"num_write":30,"num_write_kb":6528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000865"},"pg_stats":[{"pgid":"3.f","version":"201'135","reported_seq":728,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-04-24T01:58:39.141002+0000","last_change":"2024-04-24T01:58:39.141002+0000","last_active":"2024-04-24T01:58:39.141002+0000","last_peered":"2024-04-24T01:58:39.141002+0000","last_clean":"2024-04-24T01:58:39.141002+0000","last_became_active":"2024-04-24T01:55:30.386726+0000","last_became_peered":"2024-04-24T01:55:30.386726+0000","last_unstale":"2024-04-24T01:58:39.141002+0000","last_undegraded":"2024-04-24T01:58:39.141002+0000","last_fullsized":"2024-04-24T01:58:39.141002+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":11,"snaptrim_duration":0,"stat_sum":{"num_bytes":2285818,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":100,"num_read_kb":89467,"num_write":58,"num_write_kb":16446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"201'240","reported_seq":800,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-04-24T01:58:39.088552+0000","last_change":"2024-04-24T01:58:39.094751+0000","last_active":"2024-04-24T01:58:39.088552+0000","last_peered":"2024-04-24T01:58:39.088552+0000","last_clean":"2024-04-24T01:58:39.088552+0000","last_became_active":"2024-04-24T01:55:30.321866+0000","last_became_peered":"2024-04-24T01:55:30.321866+0000","last_unstale":"2024-04-24T01:58:39.088552+0000","last_undegraded":"2024-04-24T01:58:39.088552+0000","last_fullsized":"2024-04-24T01:58:39.088552+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":240,"log_dups_size":0,"ondisk_log_size":240,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":17,"snaptrim_duration":0,"stat_sum":{"num_bytes":11328116,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":135,"num_read_kb":93306,"num_write":110,"num_write_kb":29873,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"182'185","reported_seq":739,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-04-24T01:58:38.079587+0000","last_change":"2024-04-24T01:58:37.214384+0000","last_active":"2024-04-24T01:58:38.079587+0000","last_peered":"2024-04-24T01:58:38.079587+0000","last_clean":"2024-04-24T01:58:38.079587+0000","last_became_active":"2024-04-24T01:55:30.320186+0000","last_became_peered":"2024-04-24T01:55:30.320186+0000","last_unstale":"2024-04-24T01:58:38.079587+0000","last_undegraded":"2024-04-24T01:58:38.079587+0000","last_fullsized":"2024-04-24T01:58:38.079587+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":185,"log_dups_size":0,"ondisk_log_size":185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":17,"snaptrim_duration":0,"stat_sum":{"num_bytes":6353112,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":76,"num_read_kb":46994,"num_write":95,"num_write_kb":25777,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"201'214","reported_seq":785,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-04-24T01:58:39.113036+0000","last_change":"2024-04-24T01:58:39.140524+0000","last_active":"2024-04-24T01:58:39.113036+0000","last_peered":"2024-04-24T01:58:39.113036+0000","last_clean":"2024-04-24T01:58:39.113036+0000","last_became_active":"2024-04-24T01:55:30.288081+0000","last_became_peered":"2024-04-24T01:55:30.288081+0000","last_unstale":"2024-04-24T01:58:39.113036+0000","last_undegraded":"2024-04-24T01:58:39.113036+0000","last_fullsized":"2024-04-24T01:58:39.113036+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":214,"log_dups_size":0,"ondisk_log_size":214,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":21,"snaptrim_duration":0,"stat_sum":{"num_bytes":5596830,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":117,"num_read_kb":84156,"num_write":89,"num_write_kb":25019,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"182'61","reported_seq":643,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-04-24T01:58:38.251861+0000","last_change":"2024-04-24T01:58:37.092662+0000","last_active":"2024-04-24T01:58:38.251861+0000","last_peered":"2024-04-24T01:58:38.251861+0000","last_clean":"2024-04-24T01:58:38.251861+0000","last_became_active":"2024-04-24T01:55:30.333815+0000","last_became_peered":"2024-04-24T01:55:30.333815+0000","last_unstale":"2024-04-24T01:58:38.251861+0000","last_undegraded":"2024-04-24T01:58:38.251861+0000","last_fullsized":"2024-04-24T01:58:38.251861+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":5,"snaptrim_duration":0,"stat_sum":{"num_bytes":3798382,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":32,"num_read_kb":17556,"num_write":36,"num_write_kb":10196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"201'415","reported_seq":939,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-04-24T01:58:39.120800+0000","last_change":"2024-04-24T01:58:39.134556+0000","last_active":"2024-04-24T01:58:39.120800+0000","last_peered":"2024-04-24T01:58:39.120800+0000","last_clean":"2024-04-24T01:58:39.120800+0000","last_became_active":"2024-04-24T01:55:30.315075+0000","last_became_peered":"2024-04-24T01:55:30.315075+0000","last_unstale":"2024-04-24T01:58:39.120800+0000","last_undegraded":"2024-04-24T01:58:39.120800+0000","last_fullsized":"2024-04-24T01:58:39.120800+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":415,"log_dups_size":0,"ondisk_log_size":415,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":37,"snaptrim_duration":0,"stat_sum":{"num_bytes":9882648,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":172,"num_read_kb":102271,"num_write":171,"num_write_kb":45947,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"183'66","reported_seq":654,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-04-24T01:58:39.141548+0000","last_change":"2024-04-24T01:58:39.149264+0000","last_active":"2024-04-24T01:58:39.141548+0000","last_peered":"2024-04-24T01:58:39.141548+0000","last_clean":"2024-04-24T01:58:39.141548+0000","last_became_active":"2024-04-24T01:55:30.350300+0000","last_became_peered":"2024-04-24T01:55:30.350300+0000","last_unstale":"2024-04-24T01:58:39.141548+0000","last_undegraded":"2024-04-24T01:58:39.141548+0000","last_fullsized":"2024-04-24T01:58:39.141548+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0,"stat_sum":{"num_bytes":3596731,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":33,"num_read_kb":16964,"num_write":41,"num_write_kb":10355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"163'8","reported_seq":581,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-04-24T01:58:38.089801+0000","last_change":"2024-04-24T01:58:37.226831+0000","last_active":"2024-04-24T01:58:38.089801+0000","last_peered":"2024-04-24T01:58:38.089801+0000","last_clean":"2024-04-24T01:58:38.089801+0000","last_became_active":"2024-04-24T01:55:30.376813+0000","last_became_peered":"2024-04-24T01:55:30.376813+0000","last_unstale":"2024-04-24T01:58:38.089801+0000","last_undegraded":"2024-04-24T01:58:38.089801+0000","last_fullsized":"2024-04-24T01:58:38.089801+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'97","reported_seq":636,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-04-24T01:58:38.240462+0000","last_change":"2024-04-24T01:55:16.196470+0000","last_active":"2024-04-24T01:58:38.240462+0000","last_peered":"2024-04-24T01:58:38.240462+0000","last_clean":"2024-04-24T01:58:38.240462+0000","last_became_active":"2024-04-24T01:55:16.190464+0000","last_became_peered":"2024-04-24T01:55:16.190464+0000","last_unstale":"2024-04-24T01:58:38.240462+0000","last_undegraded":"2024-04-24T01:58:38.240462+0000","last_fullsized":"2024-04-24T01:58:38.240462+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:15.127115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:15.127115+0000","last_clean_scrub_stamp":"2024-04-24T01:55:15.127115+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":495,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-04-24T01:58:38.087296+0000","last_change":"2024-04-24T01:55:18.236148+0000","last_active":"2024-04-24T01:58:38.087296+0000","last_peered":"2024-04-24T01:58:38.087296+0000","last_clean":"2024-04-24T01:58:38.087296+0000","last_became_active":"2024-04-24T01:55:18.230154+0000","last_became_peered":"2024-04-24T01:55:18.230154+0000","last_unstale":"2024-04-24T01:58:38.087296+0000","last_undegraded":"2024-04-24T01:58:38.087296+0000","last_fullsized":"2024-04-24T01:58:38.087296+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"201'314","reported_seq":874,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-04-24T01:58:38.087031+0000","last_change":"2024-04-24T01:58:37.316299+0000","last_active":"2024-04-24T01:58:38.087031+0000","last_peered":"2024-04-24T01:58:38.087031+0000","last_clean":"2024-04-24T01:58:38.087031+0000","last_became_active":"2024-04-24T01:55:30.316222+0000","last_became_peered":"2024-04-24T01:55:30.316222+0000","last_unstale":"2024-04-24T01:58:38.087031+0000","last_undegraded":"2024-04-24T01:58:38.087031+0000","last_fullsized":"2024-04-24T01:58:38.087031+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":314,"log_dups_size":0,"ondisk_log_size":314,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":24,"snaptrim_duration":0,"stat_sum":{"num_bytes":10739904,"num_objects":6,"num_object_clones":2,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":162,"num_read_kb":107369,"num_write":158,"num_write_kb":44575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":508,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-04-24T01:58:39.138421+0000","last_change":"2024-04-24T01:55:18.265905+0000","last_active":"2024-04-24T01:58:39.138421+0000","last_peered":"2024-04-24T01:58:39.138421+0000","last_clean":"2024-04-24T01:58:39.138421+0000","last_became_active":"2024-04-24T01:55:18.257020+0000","last_became_peered":"2024-04-24T01:55:18.257020+0000","last_unstale":"2024-04-24T01:58:39.138421+0000","last_undegraded":"2024-04-24T01:58:39.138421+0000","last_fullsized":"2024-04-24T01:58:39.138421+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"201'264","reported_seq":807,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-04-24T01:58:38.241863+0000","last_change":"2024-04-24T01:58:37.198398+0000","last_active":"2024-04-24T01:58:38.241863+0000","last_peered":"2024-04-24T01:58:38.241863+0000","last_clean":"2024-04-24T01:58:38.241863+0000","last_became_active":"2024-04-24T01:55:30.344129+0000","last_became_peered":"2024-04-24T01:55:30.344129+0000","last_unstale":"2024-04-24T01:58:38.241863+0000","last_undegraded":"2024-04-24T01:58:38.241863+0000","last_fullsized":"2024-04-24T01:58:38.241863+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":264,"log_dups_size":0,"ondisk_log_size":264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":18,"snaptrim_duration":0,"stat_sum":{"num_bytes":6420472,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":102,"num_read_kb":60236,"num_write":114,"num_write_kb":32496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":495,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-04-24T01:58:38.078901+0000","last_change":"2024-04-24T01:55:18.215251+0000","last_active":"2024-04-24T01:58:38.078901+0000","last_peered":"2024-04-24T01:58:38.078901+0000","last_clean":"2024-04-24T01:58:38.078901+0000","last_became_active":"2024-04-24T01:55:18.206605+0000","last_became_peered":"2024-04-24T01:55:18.206605+0000","last_unstale":"2024-04-24T01:58:38.078901+0000","last_undegraded":"2024-04-24T01:58:38.078901+0000","last_fullsized":"2024-04-24T01:58:38.078901+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"201'131","reported_seq":706,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-04-24T01:58:38.253362+0000","last_change":"2024-04-24T01:58:37.129566+0000","last_active":"2024-04-24T01:58:38.253362+0000","last_peered":"2024-04-24T01:58:38.253362+0000","last_clean":"2024-04-24T01:58:38.253362+0000","last_became_active":"2024-04-24T01:55:30.401366+0000","last_became_peered":"2024-04-24T01:55:30.401366+0000","last_unstale":"2024-04-24T01:58:38.253362+0000","last_undegraded":"2024-04-24T01:58:38.253362+0000","last_fullsized":"2024-04-24T01:58:38.253362+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":131,"log_dups_size":0,"ondisk_log_size":131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":10,"snaptrim_duration":0,"stat_sum":{"num_bytes":2218590,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":82,"num_read_kb":65583,"num_write":42,"num_write_kb":10680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":508,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-04-24T01:58:39.132749+0000","last_change":"2024-04-24T01:55:18.230893+0000","last_active":"2024-04-24T01:58:39.132749+0000","last_peered":"2024-04-24T01:58:39.132749+0000","last_clean":"2024-04-24T01:58:39.132749+0000","last_became_active":"2024-04-24T01:55:18.225188+0000","last_became_peered":"2024-04-24T01:55:18.225188+0000","last_unstale":"2024-04-24T01:58:39.132749+0000","last_undegraded":"2024-04-24T01:58:39.132749+0000","last_fullsized":"2024-04-24T01:58:39.132749+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"201'216","reported_seq":762,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-04-24T01:58:39.086597+0000","last_change":"2024-04-24T01:58:39.095014+0000","last_active":"2024-04-24T01:58:39.086597+0000","last_peered":"2024-04-24T01:58:39.086597+0000","last_clean":"2024-04-24T01:58:39.086597+0000","last_became_active":"2024-04-24T01:55:30.343318+0000","last_became_peered":"2024-04-24T01:55:30.343318+0000","last_unstale":"2024-04-24T01:58:39.086597+0000","last_undegraded":"2024-04-24T01:58:39.086597+0000","last_fullsized":"2024-04-24T01:58:39.086597+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":216,"log_dups_size":0,"ondisk_log_size":216,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":16,"snaptrim_duration":0,"stat_sum":{"num_bytes":4571636,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":98,"num_read_kb":79025,"num_write":96,"num_write_kb":26701,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"201'274","reported_seq":829,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-04-24T01:58:38.089049+0000","last_change":"2024-04-24T01:58:37.264791+0000","last_active":"2024-04-24T01:58:38.089049+0000","last_peered":"2024-04-24T01:58:38.089049+0000","last_clean":"2024-04-24T01:58:38.089049+0000","last_became_active":"2024-04-24T01:55:30.345308+0000","last_became_peered":"2024-04-24T01:55:30.345308+0000","last_unstale":"2024-04-24T01:58:38.089049+0000","last_undegraded":"2024-04-24T01:58:38.089049+0000","last_fullsized":"2024-04-24T01:58:38.089049+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":274,"log_dups_size":0,"ondisk_log_size":274,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":21,"snaptrim_duration":0,"stat_sum":{"num_bytes":11109625,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":149,"num_read_kb":102577,"num_write":116,"num_write_kb":29821,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":490,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-04-24T01:58:39.096655+0000","last_change":"2024-04-24T01:55:18.224238+0000","last_active":"2024-04-24T01:58:39.096655+0000","last_peered":"2024-04-24T01:58:39.096655+0000","last_clean":"2024-04-24T01:58:39.096655+0000","last_became_active":"2024-04-24T01:55:18.219131+0000","last_became_peered":"2024-04-24T01:55:18.219131+0000","last_unstale":"2024-04-24T01:58:39.096655+0000","last_undegraded":"2024-04-24T01:58:39.096655+0000","last_fullsized":"2024-04-24T01:58:39.096655+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"174'54","reported_seq":625,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-04-24T01:58:39.095792+0000","last_change":"2024-04-24T01:58:39.103557+0000","last_active":"2024-04-24T01:58:39.095792+0000","last_peered":"2024-04-24T01:58:39.095792+0000","last_clean":"2024-04-24T01:58:39.095792+0000","last_became_active":"2024-04-24T01:55:30.319958+0000","last_became_peered":"2024-04-24T01:55:30.319958+0000","last_unstale":"2024-04-24T01:58:39.095792+0000","last_undegraded":"2024-04-24T01:58:39.095792+0000","last_fullsized":"2024-04-24T01:58:39.095792+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":5,"snaptrim_duration":0,"stat_sum":{"num_bytes":2823609,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":26,"num_read_kb":19608,"num_write":37,"num_write_kb":10363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":503,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-04-24T01:58:38.249232+0000","last_change":"2024-04-24T01:55:18.235763+0000","last_active":"2024-04-24T01:58:38.249232+0000","last_peered":"2024-04-24T01:58:38.249232+0000","last_clean":"2024-04-24T01:58:38.249232+0000","last_became_active":"2024-04-24T01:55:18.229175+0000","last_became_peered":"2024-04-24T01:55:18.229175+0000","last_unstale":"2024-04-24T01:58:38.249232+0000","last_undegraded":"2024-04-24T01:58:38.249232+0000","last_fullsized":"2024-04-24T01:58:38.249232+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"201'95","reported_seq":667,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-04-24T01:58:38.243583+0000","last_change":"2024-04-24T01:58:37.187841+0000","last_active":"2024-04-24T01:58:38.243583+0000","last_peered":"2024-04-24T01:58:38.243583+0000","last_clean":"2024-04-24T01:58:38.243583+0000","last_became_active":"2024-04-24T01:55:30.372690+0000","last_became_peered":"2024-04-24T01:55:30.372690+0000","last_unstale":"2024-04-24T01:58:38.243583+0000","last_undegraded":"2024-04-24T01:58:38.243583+0000","last_fullsized":"2024-04-24T01:58:38.243583+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":7,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":46,"num_read_kb":30459,"num_write":40,"num_write_kb":11003,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":490,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-04-24T01:58:39.097577+0000","last_change":"2024-04-24T01:55:18.278351+0000","last_active":"2024-04-24T01:58:39.097577+0000","last_peered":"2024-04-24T01:58:39.097577+0000","last_clean":"2024-04-24T01:58:39.097577+0000","last_became_active":"2024-04-24T01:55:18.273668+0000","last_became_peered":"2024-04-24T01:55:18.273668+0000","last_unstale":"2024-04-24T01:58:39.097577+0000","last_undegraded":"2024-04-24T01:58:39.097577+0000","last_fullsized":"2024-04-24T01:58:39.097577+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"201'210","reported_seq":781,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-04-24T01:58:39.099609+0000","last_change":"2024-04-24T01:58:39.104137+0000","last_active":"2024-04-24T01:58:39.099609+0000","last_peered":"2024-04-24T01:58:39.099609+0000","last_clean":"2024-04-24T01:58:39.099609+0000","last_became_active":"2024-04-24T01:55:30.363123+0000","last_became_peered":"2024-04-24T01:55:30.363123+0000","last_unstale":"2024-04-24T01:58:39.099609+0000","last_undegraded":"2024-04-24T01:58:39.099609+0000","last_fullsized":"2024-04-24T01:58:39.099609+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":210,"log_dups_size":0,"ondisk_log_size":210,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":20,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344824,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":127,"num_read_kb":102061,"num_write":112,"num_write_kb":29872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":495,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-04-24T01:58:38.090910+0000","last_change":"2024-04-24T01:55:18.241117+0000","last_active":"2024-04-24T01:58:38.090910+0000","last_peered":"2024-04-24T01:58:38.090910+0000","last_clean":"2024-04-24T01:58:38.090910+0000","last_became_active":"2024-04-24T01:55:18.234313+0000","last_became_peered":"2024-04-24T01:55:18.234313+0000","last_unstale":"2024-04-24T01:58:38.090910+0000","last_undegraded":"2024-04-24T01:58:38.090910+0000","last_fullsized":"2024-04-24T01:58:38.090910+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":88070297,"num_objects":40,"num_object_clones":5,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":1459,"num_read_kb":1017633,"num_write":1320,"num_write_kb":359972,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":179335168,"data_stored":183890608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2882,"ondisk_log_size":2882,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":1}],"osd_stats":[{"osd":3,"up_from":10,"seq":42949673164,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":86580,"kb_used_data":51436,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94285260,"statfs":{"total":96636764160,"available":96548106240,"internally_reserved":0,"allocated":52670464,"data_stored":52911661,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673164,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":70436,"kb_used_data":39452,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94301404,"statfs":{"total":96636764160,"available":96564637696,"internally_reserved":0,"allocated":40398848,"data_stored":39592160,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705869,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":73092,"kb_used_data":42108,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94298748,"statfs":{"total":96636764160,"available":96561917952,"internally_reserved":0,"allocated":43118592,"data_stored":43478779,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705868,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":77812,"kb_used_data":46828,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94294028,"statfs":{"total":96636764160,"available":96557084672,"internally_reserved":0,"allocated":47951872,"data_stored":47741952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":42131456,"data_stored":43706562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":46301184,"data_stored":47026943,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":39219200,"data_stored":40017659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":51683328,"data_stored":53139444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T01:58:41.089 INFO:tasks.rados.rados.0.smithi079.stdout:802: done (2 left) 2024-04-24T01:58:41.090 INFO:tasks.rados.rados.0.smithi079.stdout:803: read oid 42 snap -1 2024-04-24T01:58:41.090 INFO:tasks.rados.rados.0.smithi079.stdout:803: expect deleted 2024-04-24T01:58:41.090 INFO:tasks.rados.rados.0.smithi079.stdout:804: read oid 41 snap -1 2024-04-24T01:58:41.090 INFO:tasks.rados.rados.0.smithi079.stdout:804: expect (ObjNum 370 snap 90 seq_num 370) 2024-04-24T01:58:41.090 INFO:tasks.rados.rados.0.smithi079.stdout:805: read oid 1 snap -1 2024-04-24T01:58:41.090 INFO:tasks.rados.rados.0.smithi079.stdout:805: expect (ObjNum 332 snap 81 seq_num 332) 2024-04-24T01:58:41.090 INFO:tasks.rados.rados.0.smithi079.stdout:806: snap_create 2024-04-24T01:58:41.090 INFO:tasks.rados.rados.0.smithi079.stdout:799: finishing write tid 2 to smithi07929762-12 2024-04-24T01:58:41.090 INFO:tasks.rados.rados.0.smithi079.stdout:799: finishing write tid 3 to smithi07929762-12 2024-04-24T01:58:41.090 INFO:tasks.rados.rados.0.smithi079.stdout:799: finishing write tid 4 to smithi07929762-12 2024-04-24T01:58:41.090 INFO:tasks.rados.rados.0.smithi079.stdout:799: oid 12 updating version 0 to 217 2024-04-24T01:58:41.090 INFO:tasks.rados.rados.0.smithi079.stdout:799: oid 12 version 217 is already newer than 215 2024-04-24T01:58:41.091 INFO:tasks.rados.rados.0.smithi079.stdout:799: oid 12 version 217 is already newer than 217 2024-04-24T01:58:41.091 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 12 v 217 (ObjNum 391 snap 93 seq_num 391) dirty exists 2024-04-24T01:58:41.091 INFO:tasks.rados.rados.0.smithi079.stdout:799: left oid 12 (ObjNum 391 snap 93 seq_num 391) 2024-04-24T01:58:41.091 INFO:tasks.rados.rados.0.smithi079.stdout:801: expect (ObjNum 366 snap 90 seq_num 366) 2024-04-24T01:58:41.212 INFO:tasks.rados.rados.0.smithi079.stdout:805: expect (ObjNum 332 snap 81 seq_num 332) 2024-04-24T01:58:41.349 INFO:tasks.rados.rados.0.smithi079.stdout:804: expect (ObjNum 370 snap 90 seq_num 370) 2024-04-24T01:58:42.089 INFO:tasks.rados.rados.0.smithi079.stdout:799: done (5 left) 2024-04-24T01:58:42.089 INFO:tasks.rados.rados.0.smithi079.stdout:801: done (4 left) 2024-04-24T01:58:42.089 INFO:tasks.rados.rados.0.smithi079.stdout:803: done (3 left) 2024-04-24T01:58:42.090 INFO:tasks.rados.rados.0.smithi079.stdout:804: done (2 left) 2024-04-24T01:58:42.090 INFO:tasks.rados.rados.0.smithi079.stdout:805: done (1 left) 2024-04-24T01:58:42.090 INFO:tasks.rados.rados.0.smithi079.stdout:806: done (0 left) 2024-04-24T01:58:42.090 INFO:tasks.rados.rados.0.smithi079.stdout:807: delete oid 11 current snap is 94 2024-04-24T01:58:42.143 INFO:tasks.rados.rados.0.smithi079.stdout:807: done (0 left) 2024-04-24T01:58:42.143 INFO:tasks.rados.rados.0.smithi079.stdout:808: snap_remove snap 93 2024-04-24T01:58:42.145 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T01:58:42.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:58:42.266+0000 7f749859d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:42.272 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:58:42.271+0000 7f749859d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:43.094 INFO:tasks.rados.rados.0.smithi079.stdout:808: done (0 left) 2024-04-24T01:58:43.094 INFO:tasks.rados.rados.0.smithi079.stdout:809: read oid 44 snap -1 2024-04-24T01:58:43.094 INFO:tasks.rados.rados.0.smithi079.stdout:809: expect deleted 2024-04-24T01:58:43.094 INFO:tasks.rados.rados.0.smithi079.stdout:810: write oid 16 current snap is 94 2024-04-24T01:58:43.094 INFO:tasks.rados.rados.0.smithi079.stdout:810: seq_num 394 ranges {621958=782109,2089670=532288} 2024-04-24T01:58:43.101 INFO:tasks.rados.rados.0.smithi079.stdout:810: writing smithi07929762-16 from 621958 to 1404067 tid 1 2024-04-24T01:58:43.109 INFO:tasks.rados.rados.0.smithi079.stdout:810: writing smithi07929762-16 from 2089670 to 2621958 tid 2 2024-04-24T01:58:43.109 INFO:tasks.rados.rados.0.smithi079.stdout:811: read oid 24 snap -1 2024-04-24T01:58:43.109 INFO:tasks.rados.rados.0.smithi079.stdout:811: expect (ObjNum 376 snap 90 seq_num 376) 2024-04-24T01:58:43.109 INFO:tasks.rados.rados.0.smithi079.stdout:812: write oid 19 current snap is 94 2024-04-24T01:58:43.109 INFO:tasks.rados.rados.0.smithi079.stdout:812: seq_num 395 ranges {638765=457358,1748152=604803} 2024-04-24T01:58:43.114 INFO:tasks.rados.rados.0.smithi079.stdout:812: writing smithi07929762-19 from 638765 to 1096123 tid 1 2024-04-24T01:58:43.122 INFO:tasks.rados.rados.0.smithi079.stdout:812: writing smithi07929762-19 from 1748152 to 2352955 tid 2 2024-04-24T01:58:43.122 INFO:tasks.rados.rados.0.smithi079.stdout:813: read oid 42 snap -1 2024-04-24T01:58:43.122 INFO:tasks.rados.rados.0.smithi079.stdout:813: expect deleted 2024-04-24T01:58:43.122 INFO:tasks.rados.rados.0.smithi079.stdout:814: write (excl) oid 11 current snap is 94 2024-04-24T01:58:43.122 INFO:tasks.rados.rados.0.smithi079.stdout:814: seq_num 396 ranges {655572=648960,1922987=732585} 2024-04-24T01:58:43.130 INFO:tasks.rados.rados.0.smithi079.stdout:814: writing smithi07929762-11 from 655572 to 1304532 tid 1 2024-04-24T01:58:43.138 INFO:tasks.rados.rados.0.smithi079.stdout:814: writing smithi07929762-11 from 1922987 to 2655572 tid 2 2024-04-24T01:58:43.138 INFO:tasks.rados.rados.0.smithi079.stdout:815: delete oid 13 current snap is 94 2024-04-24T01:58:43.178 INFO:tasks.rados.rados.0.smithi079.stdout:810: finishing write tid 1 to smithi07929762-16 2024-04-24T01:58:43.181 INFO:tasks.rados.rados.0.smithi079.stdout:811: expect (ObjNum 376 snap 90 seq_num 376) 2024-04-24T01:58:43.286 INFO:tasks.rados.rados.0.smithi079.stdout:809: done (6 left) 2024-04-24T01:58:43.286 INFO:tasks.rados.rados.0.smithi079.stdout:811: done (5 left) 2024-04-24T01:58:43.286 INFO:tasks.rados.rados.0.smithi079.stdout:815: done (4 left) 2024-04-24T01:58:43.286 INFO:tasks.rados.rados.0.smithi079.stdout:816: delete oid 35 current snap is 94 2024-04-24T01:58:43.286 INFO:tasks.rados.rados.0.smithi079.stdout:810: finishing write tid 2 to smithi07929762-16 2024-04-24T01:58:43.286 INFO:tasks.rados.rados.0.smithi079.stdout:810: finishing write tid 3 to smithi07929762-16 2024-04-24T01:58:43.286 INFO:tasks.rados.rados.0.smithi079.stdout:810: finishing write tid 4 to smithi07929762-16 2024-04-24T01:58:43.286 INFO:tasks.rados.rados.0.smithi079.stdout:810: oid 16 updating version 0 to 418 2024-04-24T01:58:43.286 INFO:tasks.rados.rados.0.smithi079.stdout:810: oid 16 version 418 is already newer than 418 2024-04-24T01:58:43.286 INFO:tasks.rados.rados.0.smithi079.stdout:810: oid 16 version 418 is already newer than 418 2024-04-24T01:58:43.286 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 16 v 418 (ObjNum 394 snap 94 seq_num 394) dirty exists 2024-04-24T01:58:43.287 INFO:tasks.rados.rados.0.smithi079.stdout:810: left oid 16 (ObjNum 394 snap 94 seq_num 394) 2024-04-24T01:58:43.287 INFO:tasks.rados.rados.0.smithi079.stdout:812: finishing write tid 1 to smithi07929762-19 2024-04-24T01:58:43.287 INFO:tasks.rados.rados.0.smithi079.stdout:814: finishing write tid 1 to smithi07929762-11 2024-04-24T01:58:43.287 INFO:tasks.rados.rados.0.smithi079.stdout:812: finishing write tid 2 to smithi07929762-19 2024-04-24T01:58:43.287 INFO:tasks.rados.rados.0.smithi079.stdout:812: finishing write tid 3 to smithi07929762-19 2024-04-24T01:58:43.287 INFO:tasks.rados.rados.0.smithi079.stdout:812: finishing write tid 4 to smithi07929762-19 2024-04-24T01:58:43.287 INFO:tasks.rados.rados.0.smithi079.stdout:812: oid 19 updating version 0 to 267 2024-04-24T01:58:43.287 INFO:tasks.rados.rados.0.smithi079.stdout:812: oid 19 version 267 is already newer than 267 2024-04-24T01:58:43.287 INFO:tasks.rados.rados.0.smithi079.stdout:812: oid 19 version 267 is already newer than 267 2024-04-24T01:58:43.287 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 19 v 267 (ObjNum 395 snap 94 seq_num 395) dirty exists 2024-04-24T01:58:43.287 INFO:tasks.rados.rados.0.smithi079.stdout:812: left oid 19 (ObjNum 395 snap 94 seq_num 395) 2024-04-24T01:58:43.287 INFO:tasks.rados.rados.0.smithi079.stdout:814: finishing write tid 2 to smithi07929762-11 2024-04-24T01:58:43.301 INFO:tasks.rados.rados.0.smithi079.stdout:814: finishing write tid 3 to smithi07929762-11 2024-04-24T01:58:43.308 INFO:tasks.rados.rados.0.smithi079.stdout:814: finishing write tid 4 to smithi07929762-11 2024-04-24T01:58:43.308 INFO:tasks.rados.rados.0.smithi079.stdout:814: oid 11 updating version 0 to 200 2024-04-24T01:58:43.308 INFO:tasks.rados.rados.0.smithi079.stdout:814: oid 11 version 200 is already newer than 197 2024-04-24T01:58:43.308 INFO:tasks.rados.rados.0.smithi079.stdout:814: oid 11 version 200 is already newer than 196 2024-04-24T01:58:43.308 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 11 v 200 (ObjNum 396 snap 94 seq_num 396) dirty exists 2024-04-24T01:58:43.308 INFO:tasks.rados.rados.0.smithi079.stdout:814: left oid 11 (ObjNum 396 snap 94 seq_num 396) 2024-04-24T01:58:43.310 INFO:tasks.rados.rados.0.smithi079.stdout:810: done (4 left) 2024-04-24T01:58:43.310 INFO:tasks.rados.rados.0.smithi079.stdout:812: done (3 left) 2024-04-24T01:58:43.310 INFO:tasks.rados.rados.0.smithi079.stdout:813: done (2 left) 2024-04-24T01:58:43.310 INFO:tasks.rados.rados.0.smithi079.stdout:814: done (1 left) 2024-04-24T01:58:43.310 INFO:tasks.rados.rados.0.smithi079.stdout:816: done (0 left) 2024-04-24T01:58:43.310 INFO:tasks.rados.rados.0.smithi079.stdout:817: read oid 45 snap -1 2024-04-24T01:58:43.310 INFO:tasks.rados.rados.0.smithi079.stdout:817: expect (ObjNum 371 snap 90 seq_num 371) 2024-04-24T01:58:43.310 INFO:tasks.rados.rados.0.smithi079.stdout:818: delete oid 25 current snap is 94 2024-04-24T01:58:43.333 INFO:tasks.rados.rados.0.smithi079.stdout:818: done (1 left) 2024-04-24T01:58:43.333 INFO:tasks.rados.rados.0.smithi079.stdout:819: read oid 15 snap -1 2024-04-24T01:58:43.333 INFO:tasks.rados.rados.0.smithi079.stdout:819: expect deleted 2024-04-24T01:58:43.333 INFO:tasks.rados.rados.0.smithi079.stdout:820: write (excl) oid 22 current snap is 94 2024-04-24T01:58:43.333 INFO:tasks.rados.rados.0.smithi079.stdout:820: seq_num 400 ranges {722800=549956,2040562=682238} 2024-04-24T01:58:43.342 INFO:tasks.rados.rados.0.smithi079.stdout:820: writing smithi07929762-22 from 722800 to 1272756 tid 1 2024-04-24T01:58:43.348 INFO:tasks.rados.rados.0.smithi079.stdout:820: writing smithi07929762-22 from 2040562 to 2722800 tid 2 2024-04-24T01:58:43.348 INFO:tasks.rados.rados.0.smithi079.stdout:821: snap_create 2024-04-24T01:58:43.348 INFO:tasks.rados.rados.0.smithi079.stdout:817: expect (ObjNum 371 snap 90 seq_num 371) 2024-04-24T01:58:43.441 INFO:tasks.rados.rados.0.smithi079.stdout:820: finishing write tid 1 to smithi07929762-22 2024-04-24T01:58:43.441 INFO:tasks.rados.rados.0.smithi079.stdout:820: finishing write tid 2 to smithi07929762-22 2024-04-24T01:58:43.441 INFO:tasks.rados.rados.0.smithi079.stdout:820: finishing write tid 3 to smithi07929762-22 2024-04-24T01:58:43.441 INFO:tasks.rados.rados.0.smithi079.stdout:820: finishing write tid 4 to smithi07929762-22 2024-04-24T01:58:43.441 INFO:tasks.rados.rados.0.smithi079.stdout:820: oid 22 updating version 0 to 132 2024-04-24T01:58:43.441 INFO:tasks.rados.rados.0.smithi079.stdout:820: oid 22 updating version 132 to 134 2024-04-24T01:58:43.441 INFO:tasks.rados.rados.0.smithi079.stdout:820: oid 22 version 134 is already newer than 134 2024-04-24T01:58:43.441 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 22 v 134 (ObjNum 400 snap 94 seq_num 400) dirty exists 2024-04-24T01:58:43.441 INFO:tasks.rados.rados.0.smithi079.stdout:820: left oid 22 (ObjNum 400 snap 94 seq_num 400) 2024-04-24T01:58:44.099 INFO:tasks.rados.rados.0.smithi079.stdout:817: done (3 left) 2024-04-24T01:58:44.099 INFO:tasks.rados.rados.0.smithi079.stdout:819: done (2 left) 2024-04-24T01:58:44.099 INFO:tasks.rados.rados.0.smithi079.stdout:820: done (1 left) 2024-04-24T01:58:44.099 INFO:tasks.rados.rados.0.smithi079.stdout:821: done (0 left) 2024-04-24T01:58:44.099 INFO:tasks.rados.rados.0.smithi079.stdout:822: delete oid 23 current snap is 95 2024-04-24T01:58:44.167 INFO:tasks.rados.rados.0.smithi079.stdout:822: done (0 left) 2024-04-24T01:58:44.168 INFO:tasks.rados.rados.0.smithi079.stdout:823: delete oid 27 current snap is 95 2024-04-24T01:58:44.204 INFO:tasks.rados.rados.0.smithi079.stdout:823: done (0 left) 2024-04-24T01:58:44.204 INFO:tasks.rados.rados.0.smithi079.stdout:824: delete oid 18 current snap is 95 2024-04-24T01:58:44.245 INFO:tasks.rados.rados.0.smithi079.stdout:824: done (0 left) 2024-04-24T01:58:44.245 INFO:tasks.rados.rados.0.smithi079.stdout:825: delete oid 39 current snap is 95 2024-04-24T01:58:44.252 INFO:tasks.rados.rados.0.smithi079.stdout:825: done (0 left) 2024-04-24T01:58:44.252 INFO:tasks.rados.rados.0.smithi079.stdout:826: snap_remove snap 94 2024-04-24T01:58:44.373 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T01:58:44.477 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:58:44.475+0000 7fbb6766d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:44.481 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:58:44.480+0000 7fbb6766d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:45.100 INFO:tasks.rados.rados.0.smithi079.stdout:826: done (0 left) 2024-04-24T01:58:45.100 INFO:tasks.rados.rados.0.smithi079.stdout:827: read oid 6 snap -1 2024-04-24T01:58:45.100 INFO:tasks.rados.rados.0.smithi079.stdout:827: expect (ObjNum 276 snap 71 seq_num 276) 2024-04-24T01:58:45.100 INFO:tasks.rados.rados.0.smithi079.stdout:828: delete oid 47 current snap is 95 2024-04-24T01:58:45.165 INFO:tasks.rados.rados.0.smithi079.stdout:828: done (1 left) 2024-04-24T01:58:45.166 INFO:tasks.rados.rados.0.smithi079.stdout:829: write oid 25 current snap is 95 2024-04-24T01:58:45.166 INFO:tasks.rados.rados.0.smithi079.stdout:829: seq_num 406 ranges {423642=717803,1875101=766806} 2024-04-24T01:58:45.172 INFO:tasks.rados.rados.0.smithi079.stdout:829: writing smithi07929762-25 from 423642 to 1141445 tid 1 2024-04-24T01:58:45.183 INFO:tasks.rados.rados.0.smithi079.stdout:829: writing smithi07929762-25 from 1875101 to 2641907 tid 2 2024-04-24T01:58:45.183 INFO:tasks.rados.rados.0.smithi079.stdout:830: snap_create 2024-04-24T01:58:45.183 INFO:tasks.rados.rados.0.smithi079.stdout:827: expect (ObjNum 276 snap 71 seq_num 276) 2024-04-24T01:58:45.408 INFO:tasks.rados.rados.0.smithi079.stdout:829: finishing write tid 1 to smithi07929762-25 2024-04-24T01:58:45.408 INFO:tasks.rados.rados.0.smithi079.stdout:829: finishing write tid 2 to smithi07929762-25 2024-04-24T01:58:45.408 INFO:tasks.rados.rados.0.smithi079.stdout:829: finishing write tid 3 to smithi07929762-25 2024-04-24T01:58:45.408 INFO:tasks.rados.rados.0.smithi079.stdout:829: finishing write tid 4 to smithi07929762-25 2024-04-24T01:58:45.408 INFO:tasks.rados.rados.0.smithi079.stdout:829: oid 25 updating version 0 to 326 2024-04-24T01:58:45.408 INFO:tasks.rados.rados.0.smithi079.stdout:829: oid 25 updating version 326 to 328 2024-04-24T01:58:45.408 INFO:tasks.rados.rados.0.smithi079.stdout:829: oid 25 version 328 is already newer than 328 2024-04-24T01:58:45.408 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 25 v 328 (ObjNum 406 snap 95 seq_num 406) dirty exists 2024-04-24T01:58:45.408 INFO:tasks.rados.rados.0.smithi079.stdout:829: left oid 25 (ObjNum 406 snap 95 seq_num 406) 2024-04-24T01:58:46.041 INFO:tasks.rados.rados.0.smithi079.stdout:827: done (2 left) 2024-04-24T01:58:46.041 INFO:tasks.rados.rados.0.smithi079.stdout:829: done (1 left) 2024-04-24T01:58:46.041 INFO:tasks.rados.rados.0.smithi079.stdout:830: done (0 left) 2024-04-24T01:58:46.041 INFO:tasks.rados.rados.0.smithi079.stdout:831: read oid 3 snap 95 2024-04-24T01:58:46.042 INFO:tasks.rados.rados.0.smithi079.stdout:831: expect deleted 2024-04-24T01:58:46.042 INFO:tasks.rados.rados.0.smithi079.stdout:832: write oid 49 current snap is 96 2024-04-24T01:58:46.042 INFO:tasks.rados.rados.0.smithi079.stdout:832: seq_num 407 ranges {440449=793052,1933583=510464} 2024-04-24T01:58:46.048 INFO:tasks.rados.rados.0.smithi079.stdout:832: writing smithi07929762-49 from 440449 to 1233501 tid 1 2024-04-24T01:58:46.057 INFO:tasks.rados.rados.0.smithi079.stdout:832: writing smithi07929762-49 from 1933583 to 2444047 tid 2 2024-04-24T01:58:46.057 INFO:tasks.rados.rados.0.smithi079.stdout:833: read oid 17 snap -1 2024-04-24T01:58:46.057 INFO:tasks.rados.rados.0.smithi079.stdout:833: expect (ObjNum 379 snap 92 seq_num 379) 2024-04-24T01:58:46.057 INFO:tasks.rados.rados.0.smithi079.stdout:834: snap_create 2024-04-24T01:58:46.061 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-04-24T01:58:46.061 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-04-24T01:58:46.061 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-04-24T01:58:46.062 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-04-24T01:58:46.111 INFO:tasks.rados.rados.0.smithi079.stdout:833: expect (ObjNum 379 snap 92 seq_num 379) 2024-04-24T01:58:46.191 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:58:46.185+0000 7ff91b79d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:46.195 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:58:46.194+0000 7ff91b79d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:46.206 INFO:tasks.rados.rados.0.smithi079.stdout:832: finishing write tid 1 to smithi07929762-49 2024-04-24T01:58:46.206 INFO:tasks.rados.rados.0.smithi079.stdout:832: finishing write tid 2 to smithi07929762-49 2024-04-24T01:58:46.206 INFO:tasks.rados.rados.0.smithi079.stdout:832: finishing write tid 3 to smithi07929762-49 2024-04-24T01:58:46.206 INFO:tasks.rados.rados.0.smithi079.stdout:832: finishing write tid 4 to smithi07929762-49 2024-04-24T01:58:46.206 INFO:tasks.rados.rados.0.smithi079.stdout:832: oid 49 updating version 0 to 423 2024-04-24T01:58:46.206 INFO:tasks.rados.rados.0.smithi079.stdout:832: oid 49 updating version 423 to 425 2024-04-24T01:58:46.206 INFO:tasks.rados.rados.0.smithi079.stdout:832: oid 49 version 425 is already newer than 425 2024-04-24T01:58:46.206 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 49 v 425 (ObjNum 407 snap 96 seq_num 407) dirty exists 2024-04-24T01:58:46.207 INFO:tasks.rados.rados.0.smithi079.stdout:832: left oid 49 (ObjNum 407 snap 96 seq_num 407) 2024-04-24T01:58:46.209 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T01:58:46.209 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T01:58:46.209 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T01:58:46.209 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T01:58:46.209 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T01:58:46.209 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T01:58:46.209 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T01:58:46.209 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T01:58:46.209 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T01:58:46.209 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T01:58:46.209 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T01:58:46.209 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T01:58:46.212 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T01:58:46.212 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T01:58:46.212 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-04-24T01:58:46.335 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:58:46.329+0000 7efc79aad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:46.338 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:58:46.338+0000 7efc79aad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:46.353 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T01:58:46.353 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T01:58:46.353 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T01:58:46.353 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T01:58:46.353 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T01:58:46.353 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T01:58:46.353 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T01:58:46.353 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T01:58:46.353 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T01:58:46.353 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T01:58:46.353 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T01:58:46.353 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T01:58:46.357 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T01:58:46.357 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T01:58:46.358 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-04-24T01:58:46.466 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:58:46.460+0000 7f0519c2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:46.470 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:58:46.469+0000 7f0519c2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:46.486 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T01:58:46.486 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T01:58:46.486 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T01:58:46.486 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T01:58:46.486 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T01:58:46.486 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T01:58:46.486 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T01:58:46.486 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T01:58:46.486 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T01:58:46.487 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T01:58:46.487 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T01:58:46.487 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T01:58:46.489 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T01:58:46.490 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T01:58:46.490 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-04-24T01:58:46.573 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T01:58:46.598 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:58:46.592+0000 7fc2b959d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:46.602 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:58:46.602+0000 7fc2b959d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:46.618 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T01:58:46.618 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T01:58:46.618 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T01:58:46.618 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T01:58:46.619 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T01:58:46.619 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T01:58:46.619 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T01:58:46.619 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T01:58:46.619 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T01:58:46.619 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T01:58:46.619 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T01:58:46.619 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T01:58:46.622 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T01:58:46.622 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T01:58:46.680 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:58:46.678+0000 7f0c5d82d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:46.684 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:58:46.684+0000 7f0c5d82d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:47.042 INFO:tasks.rados.rados.0.smithi079.stdout:831: done (3 left) 2024-04-24T01:58:47.042 INFO:tasks.rados.rados.0.smithi079.stdout:832: done (2 left) 2024-04-24T01:58:47.042 INFO:tasks.rados.rados.0.smithi079.stdout:833: done (1 left) 2024-04-24T01:58:47.042 INFO:tasks.rados.rados.0.smithi079.stdout:834: done (0 left) 2024-04-24T01:58:47.043 INFO:tasks.rados.rados.0.smithi079.stdout:835: read oid 1 snap -1 2024-04-24T01:58:47.043 INFO:tasks.rados.rados.0.smithi079.stdout:835: expect (ObjNum 332 snap 81 seq_num 332) 2024-04-24T01:58:47.043 INFO:tasks.rados.rados.0.smithi079.stdout:836: read oid 43 snap -1 2024-04-24T01:58:47.043 INFO:tasks.rados.rados.0.smithi079.stdout:836: expect deleted 2024-04-24T01:58:47.043 INFO:tasks.rados.rados.0.smithi079.stdout:837: read oid 41 snap -1 2024-04-24T01:58:47.043 INFO:tasks.rados.rados.0.smithi079.stdout:837: expect (ObjNum 370 snap 90 seq_num 370) 2024-04-24T01:58:47.043 INFO:tasks.rados.rados.0.smithi079.stdout:838: write (excl) oid 25 current snap is 97 2024-04-24T01:58:47.043 INFO:tasks.rados.rados.0.smithi079.stdout:838: seq_num 408 ranges {457256=468301,1592065=770475,2854581=2675} 2024-04-24T01:58:47.049 INFO:tasks.rados.rados.0.smithi079.stdout:838: writing smithi07929762-25 from 457256 to 925557 tid 1 2024-04-24T01:58:47.058 INFO:tasks.rados.rados.0.smithi079.stdout:838: writing smithi07929762-25 from 1592065 to 2362540 tid 2 2024-04-24T01:58:47.061 INFO:tasks.rados.rados.0.smithi079.stdout:838: writing smithi07929762-25 from 2854581 to 2857256 tid 3 2024-04-24T01:58:47.061 INFO:tasks.rados.rados.0.smithi079.stdout:839: delete oid 13 current snap is 97 2024-04-24T01:58:47.096 INFO:tasks.rados.rados.0.smithi079.stdout:837: expect (ObjNum 370 snap 90 seq_num 370) 2024-04-24T01:58:47.190 INFO:tasks.rados.rados.0.smithi079.stdout:835: expect (ObjNum 332 snap 81 seq_num 332) 2024-04-24T01:58:47.325 INFO:tasks.rados.rados.0.smithi079.stdout:838: finishing write tid 1 to smithi07929762-25 2024-04-24T01:58:47.325 INFO:tasks.rados.rados.0.smithi079.stdout:838: finishing write tid 2 to smithi07929762-25 2024-04-24T01:58:47.325 INFO:tasks.rados.rados.0.smithi079.stdout:838: finishing write tid 3 to smithi07929762-25 2024-04-24T01:58:47.325 INFO:tasks.rados.rados.0.smithi079.stdout:838: finishing write tid 4 to smithi07929762-25 2024-04-24T01:58:47.325 INFO:tasks.rados.rados.0.smithi079.stdout:838: finishing write tid 5 to smithi07929762-25 2024-04-24T01:58:47.325 INFO:tasks.rados.rados.0.smithi079.stdout:838: oid 25 updating version 0 to 330 2024-04-24T01:58:47.325 INFO:tasks.rados.rados.0.smithi079.stdout:838: oid 25 updating version 330 to 333 2024-04-24T01:58:47.325 INFO:tasks.rados.rados.0.smithi079.stdout:838: oid 25 version 333 is already newer than 333 2024-04-24T01:58:47.325 INFO:tasks.rados.rados.0.smithi079.stdout:838: oid 25 version 333 is already newer than 333 2024-04-24T01:58:47.325 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 25 v 333 (ObjNum 408 snap 97 seq_num 408) dirty exists 2024-04-24T01:58:47.325 INFO:tasks.rados.rados.0.smithi079.stdout:838: left oid 25 (ObjNum 408 snap 97 seq_num 408) 2024-04-24T01:58:47.326 INFO:tasks.rados.rados.0.smithi079.stdout:835: done (4 left) 2024-04-24T01:58:47.326 INFO:tasks.rados.rados.0.smithi079.stdout:836: done (3 left) 2024-04-24T01:58:47.326 INFO:tasks.rados.rados.0.smithi079.stdout:837: done (2 left) 2024-04-24T01:58:47.326 INFO:tasks.rados.rados.0.smithi079.stdout:838: done (1 left) 2024-04-24T01:58:47.326 INFO:tasks.rados.rados.0.smithi079.stdout:839: done (0 left) 2024-04-24T01:58:47.326 INFO:tasks.rados.rados.0.smithi079.stdout:840: read oid 22 snap 95 2024-04-24T01:58:47.326 INFO:tasks.rados.rados.0.smithi079.stdout:840: expect (ObjNum 400 snap 94 seq_num 400) 2024-04-24T01:58:47.326 INFO:tasks.rados.rados.0.smithi079.stdout:841: read oid 6 snap -1 2024-04-24T01:58:47.326 INFO:tasks.rados.rados.0.smithi079.stdout:841: expect (ObjNum 276 snap 71 seq_num 276) 2024-04-24T01:58:47.326 INFO:tasks.rados.rados.0.smithi079.stdout:842: read oid 24 snap 96 2024-04-24T01:58:47.326 INFO:tasks.rados.rados.0.smithi079.stdout:842: expect (ObjNum 376 snap 90 seq_num 376) 2024-04-24T01:58:47.326 INFO:tasks.rados.rados.0.smithi079.stdout:843: delete oid 28 current snap is 97 2024-04-24T01:58:47.347 INFO:tasks.rados.rados.0.smithi079.stdout:840: expect (ObjNum 400 snap 94 seq_num 400) 2024-04-24T01:58:47.451 INFO:tasks.rados.rados.0.smithi079.stdout:842: expect (ObjNum 376 snap 90 seq_num 376) 2024-04-24T01:58:47.539 INFO:tasks.rados.rados.0.smithi079.stdout:841: expect (ObjNum 276 snap 71 seq_num 276) 2024-04-24T01:58:47.752 INFO:tasks.rados.rados.0.smithi079.stdout:840: done (3 left) 2024-04-24T01:58:47.752 INFO:tasks.rados.rados.0.smithi079.stdout:841: done (2 left) 2024-04-24T01:58:47.752 INFO:tasks.rados.rados.0.smithi079.stdout:842: done (1 left) 2024-04-24T01:58:47.753 INFO:tasks.rados.rados.0.smithi079.stdout:843: done (0 left) 2024-04-24T01:58:47.753 INFO:tasks.rados.rados.0.smithi079.stdout:844: snap_remove snap 95 2024-04-24T01:58:48.106 INFO:tasks.rados.rados.0.smithi079.stdout:844: done (0 left) 2024-04-24T01:58:48.106 INFO:tasks.rados.rados.0.smithi079.stdout:845: write oid 45 current snap is 97 2024-04-24T01:58:48.106 INFO:tasks.rados.rados.0.smithi079.stdout:845: seq_num 411 ranges {507677=410401,1367511=517802,2420497=487180} 2024-04-24T01:58:48.110 INFO:tasks.rados.rados.0.smithi079.stdout:845: writing smithi07929762-45 from 507677 to 918078 tid 1 2024-04-24T01:58:48.116 INFO:tasks.rados.rados.0.smithi079.stdout:845: writing smithi07929762-45 from 1367511 to 1885313 tid 2 2024-04-24T01:58:48.121 INFO:tasks.rados.rados.0.smithi079.stdout:845: writing smithi07929762-45 from 2420497 to 2907677 tid 3 2024-04-24T01:58:48.121 INFO:tasks.rados.rados.0.smithi079.stdout:846: write oid 46 current snap is 97 2024-04-24T01:58:48.121 INFO:tasks.rados.rados.0.smithi079.stdout:846: seq_num 412 ranges {524484=485650,1425993=661460,2847920=76564} 2024-04-24T01:58:48.127 INFO:tasks.rados.rados.0.smithi079.stdout:846: writing smithi07929762-46 from 524484 to 1010134 tid 1 2024-04-24T01:58:48.133 INFO:tasks.rados.rados.0.smithi079.stdout:846: writing smithi07929762-46 from 1425993 to 2087453 tid 2 2024-04-24T01:58:48.139 INFO:tasks.rados.rados.0.smithi079.stdout:846: writing smithi07929762-46 from 2847920 to 2924484 tid 3 2024-04-24T01:58:48.139 INFO:tasks.rados.rados.0.smithi079.stdout:847: read oid 13 snap 96 2024-04-24T01:58:48.139 INFO:tasks.rados.rados.0.smithi079.stdout:847: expect deleted 2024-04-24T01:58:48.139 INFO:tasks.rados.rados.0.smithi079.stdout:848: write (excl) oid 1 current snap is 97 2024-04-24T01:58:48.139 INFO:tasks.rados.rados.0.smithi079.stdout:848: seq_num 413 ranges {541291=560899,1884475=521471} 2024-04-24T01:58:48.145 INFO:tasks.rados.rados.0.smithi079.stdout:848: writing smithi07929762-1 from 541291 to 1102190 tid 1 2024-04-24T01:58:48.155 INFO:tasks.rados.rados.0.smithi079.stdout:848: writing smithi07929762-1 from 1884475 to 2405946 tid 2 2024-04-24T01:58:48.155 INFO:tasks.rados.rados.0.smithi079.stdout:849: read oid 19 snap 96 2024-04-24T01:58:48.155 INFO:tasks.rados.rados.0.smithi079.stdout:849: expect (ObjNum 395 snap 94 seq_num 395) 2024-04-24T01:58:48.155 INFO:tasks.rados.rados.0.smithi079.stdout:850: write (excl) oid 42 current snap is 97 2024-04-24T01:58:48.155 INFO:tasks.rados.rados.0.smithi079.stdout:850: seq_num 414 ranges {558098=636148,1826604=665129} 2024-04-24T01:58:48.165 INFO:tasks.rados.rados.0.smithi079.stdout:850: writing smithi07929762-42 from 558098 to 1194246 tid 1 2024-04-24T01:58:48.175 INFO:tasks.rados.rados.0.smithi079.stdout:850: writing smithi07929762-42 from 1826604 to 2491733 tid 2 2024-04-24T01:58:48.175 INFO:tasks.rados.rados.0.smithi079.stdout:851: delete oid 29 current snap is 97 2024-04-24T01:58:48.212 INFO:tasks.rados.rados.0.smithi079.stdout:850: finishing write tid 1 to smithi07929762-42 2024-04-24T01:58:48.219 INFO:tasks.rados.rados.0.smithi079.stdout:845: finishing write tid 1 to smithi07929762-45 2024-04-24T01:58:48.225 INFO:tasks.rados.rados.0.smithi079.stdout:849: expect (ObjNum 395 snap 94 seq_num 395) 2024-04-24T01:58:48.396 INFO:tasks.rados.rados.0.smithi079.stdout:845: finishing write tid 2 to smithi07929762-45 2024-04-24T01:58:48.396 INFO:tasks.rados.rados.0.smithi079.stdout:845: finishing write tid 3 to smithi07929762-45 2024-04-24T01:58:48.396 INFO:tasks.rados.rados.0.smithi079.stdout:845: finishing write tid 4 to smithi07929762-45 2024-04-24T01:58:48.396 INFO:tasks.rados.rados.0.smithi079.stdout:845: finishing write tid 5 to smithi07929762-45 2024-04-24T01:58:48.396 INFO:tasks.rados.rados.0.smithi079.stdout:845: oid 45 updating version 0 to 249 2024-04-24T01:58:48.396 INFO:tasks.rados.rados.0.smithi079.stdout:845: oid 45 version 249 is already newer than 249 2024-04-24T01:58:48.396 INFO:tasks.rados.rados.0.smithi079.stdout:845: oid 45 version 249 is already newer than 249 2024-04-24T01:58:48.396 INFO:tasks.rados.rados.0.smithi079.stdout:845: oid 45 version 249 is already newer than 249 2024-04-24T01:58:48.397 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 45 v 249 (ObjNum 411 snap 97 seq_num 411) dirty exists 2024-04-24T01:58:48.397 INFO:tasks.rados.rados.0.smithi079.stdout:845: left oid 45 (ObjNum 411 snap 97 seq_num 411) 2024-04-24T01:58:48.397 INFO:tasks.rados.rados.0.smithi079.stdout:846: finishing write tid 1 to smithi07929762-46 2024-04-24T01:58:48.397 INFO:tasks.rados.rados.0.smithi079.stdout:845: done (6 left) 2024-04-24T01:58:48.397 INFO:tasks.rados.rados.0.smithi079.stdout:847: done (5 left) 2024-04-24T01:58:48.397 INFO:tasks.rados.rados.0.smithi079.stdout:849: done (4 left) 2024-04-24T01:58:48.397 INFO:tasks.rados.rados.0.smithi079.stdout:851: done (3 left) 2024-04-24T01:58:48.397 INFO:tasks.rados.rados.0.smithi079.stdout:852: delete oid 43 current snap is 97 2024-04-24T01:58:48.397 INFO:tasks.rados.rados.0.smithi079.stdout:850: finishing write tid 2 to smithi07929762-42 2024-04-24T01:58:48.398 INFO:tasks.rados.rados.0.smithi079.stdout:850: finishing write tid 3 to smithi07929762-42 2024-04-24T01:58:48.398 INFO:tasks.rados.rados.0.smithi079.stdout:846: finishing write tid 2 to smithi07929762-46 2024-04-24T01:58:48.398 INFO:tasks.rados.rados.0.smithi079.stdout:846: finishing write tid 3 to smithi07929762-46 2024-04-24T01:58:48.398 INFO:tasks.rados.rados.0.smithi079.stdout:846: finishing write tid 4 to smithi07929762-46 2024-04-24T01:58:48.398 INFO:tasks.rados.rados.0.smithi079.stdout:846: finishing write tid 5 to smithi07929762-46 2024-04-24T01:58:48.398 INFO:tasks.rados.rados.0.smithi079.stdout:846: oid 46 updating version 0 to 218 2024-04-24T01:58:48.398 INFO:tasks.rados.rados.0.smithi079.stdout:846: oid 46 version 218 is already newer than 218 2024-04-24T01:58:48.398 INFO:tasks.rados.rados.0.smithi079.stdout:846: oid 46 version 218 is already newer than 218 2024-04-24T01:58:48.399 INFO:tasks.rados.rados.0.smithi079.stdout:846: oid 46 version 218 is already newer than 218 2024-04-24T01:58:48.399 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 46 v 218 (ObjNum 412 snap 97 seq_num 412) dirty exists 2024-04-24T01:58:48.399 INFO:tasks.rados.rados.0.smithi079.stdout:846: left oid 46 (ObjNum 412 snap 97 seq_num 412) 2024-04-24T01:58:48.399 INFO:tasks.rados.rados.0.smithi079.stdout:850: finishing write tid 4 to smithi07929762-42 2024-04-24T01:58:48.399 INFO:tasks.rados.rados.0.smithi079.stdout:850: oid 42 updating version 0 to 203 2024-04-24T01:58:48.400 INFO:tasks.rados.rados.0.smithi079.stdout:850: oid 42 version 203 is already newer than 203 2024-04-24T01:58:48.400 INFO:tasks.rados.rados.0.smithi079.stdout:850: oid 42 version 203 is already newer than 201 2024-04-24T01:58:48.400 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 42 v 203 (ObjNum 414 snap 97 seq_num 414) dirty exists 2024-04-24T01:58:48.400 INFO:tasks.rados.rados.0.smithi079.stdout:850: left oid 42 (ObjNum 414 snap 97 seq_num 414) 2024-04-24T01:58:48.400 INFO:tasks.rados.rados.0.smithi079.stdout:848: finishing write tid 1 to smithi07929762-1 2024-04-24T01:58:48.400 INFO:tasks.rados.rados.0.smithi079.stdout:848: finishing write tid 2 to smithi07929762-1 2024-04-24T01:58:48.400 INFO:tasks.rados.rados.0.smithi079.stdout:848: finishing write tid 3 to smithi07929762-1 2024-04-24T01:58:48.400 INFO:tasks.rados.rados.0.smithi079.stdout:848: finishing write tid 4 to smithi07929762-1 2024-04-24T01:58:48.400 INFO:tasks.rados.rados.0.smithi079.stdout:848: oid 1 updating version 0 to 429 2024-04-24T01:58:48.400 INFO:tasks.rados.rados.0.smithi079.stdout:848: oid 1 version 429 is already newer than 427 2024-04-24T01:58:48.400 INFO:tasks.rados.rados.0.smithi079.stdout:848: oid 1 version 429 is already newer than 429 2024-04-24T01:58:48.400 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 1 v 429 (ObjNum 413 snap 97 seq_num 413) dirty exists 2024-04-24T01:58:48.401 INFO:tasks.rados.rados.0.smithi079.stdout:848: left oid 1 (ObjNum 413 snap 97 seq_num 413) 2024-04-24T01:58:48.401 INFO:tasks.rados.rados.0.smithi079.stdout:846: done (3 left) 2024-04-24T01:58:48.401 INFO:tasks.rados.rados.0.smithi079.stdout:848: done (2 left) 2024-04-24T01:58:48.401 INFO:tasks.rados.rados.0.smithi079.stdout:850: done (1 left) 2024-04-24T01:58:48.401 INFO:tasks.rados.rados.0.smithi079.stdout:852: done (0 left) 2024-04-24T01:58:48.401 INFO:tasks.rados.rados.0.smithi079.stdout:853: delete oid 31 current snap is 97 2024-04-24T01:58:48.410 INFO:tasks.rados.rados.0.smithi079.stdout:853: done (0 left) 2024-04-24T01:58:48.410 INFO:tasks.rados.rados.0.smithi079.stdout:854: read oid 22 snap -1 2024-04-24T01:58:48.410 INFO:tasks.rados.rados.0.smithi079.stdout:854: expect (ObjNum 400 snap 94 seq_num 400) 2024-04-24T01:58:48.411 INFO:tasks.rados.rados.0.smithi079.stdout:855: snap_remove snap 96 2024-04-24T01:58:48.776 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T01:58:48.882 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:58:48.880+0000 7efc7cf9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:48.886 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:58:48.885+0000 7efc7cf9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:49.114 INFO:tasks.rados.rados.0.smithi079.stdout:855: done (1 left) 2024-04-24T01:58:49.114 INFO:tasks.rados.rados.0.smithi079.stdout:856: delete oid 3 current snap is 97 2024-04-24T01:58:49.114 INFO:tasks.rados.rados.0.smithi079.stdout:854: expect (ObjNum 400 snap 94 seq_num 400) 2024-04-24T01:58:49.235 INFO:tasks.rados.rados.0.smithi079.stdout:854: done (1 left) 2024-04-24T01:58:49.235 INFO:tasks.rados.rados.0.smithi079.stdout:856: done (0 left) 2024-04-24T01:58:49.235 INFO:tasks.rados.rados.0.smithi079.stdout:857: delete oid 16 current snap is 97 2024-04-24T01:58:49.253 INFO:tasks.rados.rados.0.smithi079.stdout:857: done (0 left) 2024-04-24T01:58:49.253 INFO:tasks.rados.rados.0.smithi079.stdout:858: write oid 4 current snap is 97 2024-04-24T01:58:49.253 INFO:tasks.rados.rados.0.smithi079.stdout:858: seq_num 420 ranges {658940=403995,1777496=676136,2950951=107989} 2024-04-24T01:58:49.259 INFO:tasks.rados.rados.0.smithi079.stdout:858: writing smithi07929762-4 from 658940 to 1062935 tid 1 2024-04-24T01:58:49.270 INFO:tasks.rados.rados.0.smithi079.stdout:858: writing smithi07929762-4 from 1777496 to 2453632 tid 2 2024-04-24T01:58:49.271 INFO:tasks.rados.rados.0.smithi079.stdout:858: writing smithi07929762-4 from 2950951 to 3058940 tid 3 2024-04-24T01:58:49.271 INFO:tasks.rados.rados.0.smithi079.stdout:859: read oid 11 snap -1 2024-04-24T01:58:49.271 INFO:tasks.rados.rados.0.smithi079.stdout:859: expect (ObjNum 396 snap 94 seq_num 396) 2024-04-24T01:58:49.271 INFO:tasks.rados.rados.0.smithi079.stdout:860: read oid 44 snap -1 2024-04-24T01:58:49.271 INFO:tasks.rados.rados.0.smithi079.stdout:860: expect deleted 2024-04-24T01:58:49.271 INFO:tasks.rados.rados.0.smithi079.stdout:861: delete oid 40 current snap is 97 2024-04-24T01:58:49.290 INFO:tasks.rados.rados.0.smithi079.stdout:858: finishing write tid 1 to smithi07929762-4 2024-04-24T01:58:49.307 INFO:tasks.rados.rados.0.smithi079.stdout:858: finishing write tid 2 to smithi07929762-4 2024-04-24T01:58:49.308 INFO:tasks.rados.rados.0.smithi079.stdout:860: done (3 left) 2024-04-24T01:58:49.308 INFO:tasks.rados.rados.0.smithi079.stdout:861: done (2 left) 2024-04-24T01:58:49.308 INFO:tasks.rados.rados.0.smithi079.stdout:862: read oid 45 snap -1 2024-04-24T01:58:49.309 INFO:tasks.rados.rados.0.smithi079.stdout:862: expect (ObjNum 411 snap 97 seq_num 411) 2024-04-24T01:58:49.309 INFO:tasks.rados.rados.0.smithi079.stdout:863: write (excl) oid 3 current snap is 97 2024-04-24T01:58:49.309 INFO:tasks.rados.rados.0.smithi079.stdout:863: seq_num 422 ranges {692554=554493,1778107=679805,2889444=203110} 2024-04-24T01:58:49.315 INFO:tasks.rados.rados.0.smithi079.stdout:863: writing smithi07929762-3 from 692554 to 1247047 tid 1 2024-04-24T01:58:49.322 INFO:tasks.rados.rados.0.smithi079.stdout:863: writing smithi07929762-3 from 1778107 to 2457912 tid 2 2024-04-24T01:58:49.327 INFO:tasks.rados.rados.0.smithi079.stdout:863: writing smithi07929762-3 from 2889444 to 3092554 tid 3 2024-04-24T01:58:49.327 INFO:tasks.rados.rados.0.smithi079.stdout:864: read oid 38 snap -1 2024-04-24T01:58:49.327 INFO:tasks.rados.rados.0.smithi079.stdout:864: expect (ObjNum 390 snap 93 seq_num 390) 2024-04-24T01:58:49.327 INFO:tasks.rados.rados.0.smithi079.stdout:865: snap_create 2024-04-24T01:58:49.327 INFO:tasks.rados.rados.0.smithi079.stdout:859: expect (ObjNum 396 snap 94 seq_num 396) 2024-04-24T01:58:49.431 INFO:tasks.rados.rados.0.smithi079.stdout:858: finishing write tid 3 to smithi07929762-4 2024-04-24T01:58:49.432 INFO:tasks.rados.rados.0.smithi079.stdout:858: finishing write tid 4 to smithi07929762-4 2024-04-24T01:58:49.432 INFO:tasks.rados.rados.0.smithi079.stdout:858: finishing write tid 5 to smithi07929762-4 2024-04-24T01:58:49.432 INFO:tasks.rados.rados.0.smithi079.stdout:858: oid 4 updating version 0 to 74 2024-04-24T01:58:49.432 INFO:tasks.rados.rados.0.smithi079.stdout:858: oid 4 version 74 is already newer than 73 2024-04-24T01:58:49.432 INFO:tasks.rados.rados.0.smithi079.stdout:858: oid 4 version 74 is already newer than 74 2024-04-24T01:58:49.432 INFO:tasks.rados.rados.0.smithi079.stdout:858: oid 4 version 74 is already newer than 74 2024-04-24T01:58:49.432 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 4 v 74 (ObjNum 420 snap 97 seq_num 420) dirty exists 2024-04-24T01:58:49.433 INFO:tasks.rados.rados.0.smithi079.stdout:858: left oid 4 (ObjNum 420 snap 97 seq_num 420) 2024-04-24T01:58:49.433 INFO:tasks.rados.rados.0.smithi079.stdout:862: expect (ObjNum 411 snap 97 seq_num 411) 2024-04-24T01:58:49.568 INFO:tasks.rados.rados.0.smithi079.stdout:864: expect (ObjNum 390 snap 93 seq_num 390) 2024-04-24T01:58:49.663 INFO:tasks.rados.rados.0.smithi079.stdout:863: finishing write tid 1 to smithi07929762-3 2024-04-24T01:58:49.663 INFO:tasks.rados.rados.0.smithi079.stdout:863: finishing write tid 2 to smithi07929762-3 2024-04-24T01:58:49.663 INFO:tasks.rados.rados.0.smithi079.stdout:863: finishing write tid 3 to smithi07929762-3 2024-04-24T01:58:49.664 INFO:tasks.rados.rados.0.smithi079.stdout:863: finishing write tid 4 to smithi07929762-3 2024-04-24T01:58:49.664 INFO:tasks.rados.rados.0.smithi079.stdout:863: finishing write tid 5 to smithi07929762-3 2024-04-24T01:58:49.664 INFO:tasks.rados.rados.0.smithi079.stdout:863: oid 3 updating version 0 to 435 2024-04-24T01:58:49.664 INFO:tasks.rados.rados.0.smithi079.stdout:863: oid 3 updating version 435 to 438 2024-04-24T01:58:49.664 INFO:tasks.rados.rados.0.smithi079.stdout:863: oid 3 version 438 is already newer than 438 2024-04-24T01:58:49.664 INFO:tasks.rados.rados.0.smithi079.stdout:863: oid 3 version 438 is already newer than 438 2024-04-24T01:58:49.664 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 3 v 438 (ObjNum 422 snap 97 seq_num 422) dirty exists 2024-04-24T01:58:49.665 INFO:tasks.rados.rados.0.smithi079.stdout:863: left oid 3 (ObjNum 422 snap 97 seq_num 422) 2024-04-24T01:58:50.119 INFO:tasks.rados.rados.0.smithi079.stdout:858: done (5 left) 2024-04-24T01:58:50.119 INFO:tasks.rados.rados.0.smithi079.stdout:859: done (4 left) 2024-04-24T01:58:50.120 INFO:tasks.rados.rados.0.smithi079.stdout:862: done (3 left) 2024-04-24T01:58:50.120 INFO:tasks.rados.rados.0.smithi079.stdout:863: done (2 left) 2024-04-24T01:58:50.120 INFO:tasks.rados.rados.0.smithi079.stdout:864: done (1 left) 2024-04-24T01:58:50.120 INFO:tasks.rados.rados.0.smithi079.stdout:865: done (0 left) 2024-04-24T01:58:50.120 INFO:tasks.rados.rados.0.smithi079.stdout:866: delete oid 17 current snap is 98 2024-04-24T01:58:50.175 INFO:tasks.rados.rados.0.smithi079.stdout:866: done (0 left) 2024-04-24T01:58:50.175 INFO:tasks.rados.rados.0.smithi079.stdout:867: read oid 49 snap -1 2024-04-24T01:58:50.175 INFO:tasks.rados.rados.0.smithi079.stdout:867: expect (ObjNum 407 snap 96 seq_num 407) 2024-04-24T01:58:50.175 INFO:tasks.rados.rados.0.smithi079.stdout:868: delete oid 28 current snap is 98 2024-04-24T01:58:50.183 INFO:tasks.rados.rados.0.smithi079.stdout:868: done (1 left) 2024-04-24T01:58:50.183 INFO:tasks.rados.rados.0.smithi079.stdout:869: read oid 13 snap -1 2024-04-24T01:58:50.183 INFO:tasks.rados.rados.0.smithi079.stdout:869: expect deleted 2024-04-24T01:58:50.183 INFO:tasks.rados.rados.0.smithi079.stdout:870: write oid 34 current snap is 98 2024-04-24T01:58:50.183 INFO:tasks.rados.rados.0.smithi079.stdout:870: seq_num 425 ranges {742975=780240,1953553=427132,2971713=171262} 2024-04-24T01:58:50.192 INFO:tasks.rados.rados.0.smithi079.stdout:870: writing smithi07929762-34 from 742975 to 1523215 tid 1 2024-04-24T01:58:50.197 INFO:tasks.rados.rados.0.smithi079.stdout:870: writing smithi07929762-34 from 1953553 to 2380685 tid 2 2024-04-24T01:58:50.202 INFO:tasks.rados.rados.0.smithi079.stdout:870: writing smithi07929762-34 from 2971713 to 3142975 tid 3 2024-04-24T01:58:50.202 INFO:tasks.rados.rados.0.smithi079.stdout:871: write (excl) oid 9 current snap is 98 2024-04-24T01:58:50.203 INFO:tasks.rados.rados.0.smithi079.stdout:871: seq_num 426 ranges {759782=571842,2012035=687143} 2024-04-24T01:58:50.213 INFO:tasks.rados.rados.0.smithi079.stdout:871: writing smithi07929762-9 from 759782 to 1331624 tid 1 2024-04-24T01:58:50.222 INFO:tasks.rados.rados.0.smithi079.stdout:871: writing smithi07929762-9 from 2012035 to 2699178 tid 2 2024-04-24T01:58:50.223 INFO:tasks.rados.rados.0.smithi079.stdout:872: write oid 50 current snap is 98 2024-04-24T01:58:50.223 INFO:tasks.rados.rados.0.smithi079.stdout:872: seq_num 427 ranges {776589=647091,2070517=430801,3026559=150030} 2024-04-24T01:58:50.231 INFO:tasks.rados.rados.0.smithi079.stdout:872: writing smithi07929762-50 from 776589 to 1423680 tid 1 2024-04-24T01:58:50.238 INFO:tasks.rados.rados.0.smithi079.stdout:872: writing smithi07929762-50 from 2070517 to 2501318 tid 2 2024-04-24T01:58:50.243 INFO:tasks.rados.rados.0.smithi079.stdout:872: writing smithi07929762-50 from 3026559 to 3176589 tid 3 2024-04-24T01:58:50.243 INFO:tasks.rados.rados.0.smithi079.stdout:873: write (excl) oid 20 current snap is 98 2024-04-24T01:58:50.244 INFO:tasks.rados.rados.0.smithi079.stdout:873: seq_num 428 ranges {793396=722340,2128999=690812} 2024-04-24T01:58:50.255 INFO:tasks.rados.rados.0.smithi079.stdout:873: writing smithi07929762-20 from 793396 to 1515736 tid 1 2024-04-24T01:58:50.263 INFO:tasks.rados.rados.0.smithi079.stdout:873: writing smithi07929762-20 from 2128999 to 2819811 tid 2 2024-04-24T01:58:50.263 INFO:tasks.rados.rados.0.smithi079.stdout:874: write (excl) oid 42 current snap is 98 2024-04-24T01:58:50.263 INFO:tasks.rados.rados.0.smithi079.stdout:874: seq_num 429 ranges {410203=797589,1787481=434470,2681405=528798} 2024-04-24T01:58:50.269 INFO:tasks.rados.rados.0.smithi079.stdout:874: writing smithi07929762-42 from 410203 to 1207792 tid 1 2024-04-24T01:58:50.275 INFO:tasks.rados.rados.0.smithi079.stdout:874: writing smithi07929762-42 from 1787481 to 2221951 tid 2 2024-04-24T01:58:50.283 INFO:tasks.rados.rados.0.smithi079.stdout:874: writing smithi07929762-42 from 2681405 to 3210203 tid 3 2024-04-24T01:58:50.283 INFO:tasks.rados.rados.0.smithi079.stdout:869: done (6 left) 2024-04-24T01:58:50.283 INFO:tasks.rados.rados.0.smithi079.stdout:875: write oid 28 current snap is 98 2024-04-24T01:58:50.283 INFO:tasks.rados.rados.0.smithi079.stdout:871: finishing write tid 1 to smithi07929762-9 2024-04-24T01:58:50.283 INFO:tasks.rados.rados.0.smithi079.stdout:875: seq_num 430 ranges {427010=472838,1329610=694481,2708828=518182} 2024-04-24T01:58:50.288 INFO:tasks.rados.rados.0.smithi079.stdout:875: writing smithi07929762-28 from 427010 to 899848 tid 1 2024-04-24T01:58:50.294 INFO:tasks.rados.rados.0.smithi079.stdout:875: writing smithi07929762-28 from 1329610 to 2024091 tid 2 2024-04-24T01:58:50.303 INFO:tasks.rados.rados.0.smithi079.stdout:875: writing smithi07929762-28 from 2708828 to 3227010 tid 3 2024-04-24T01:58:50.303 INFO:tasks.rados.rados.0.smithi079.stdout:876: read oid 18 snap -1 2024-04-24T01:58:50.303 INFO:tasks.rados.rados.0.smithi079.stdout:876: expect deleted 2024-04-24T01:58:50.303 INFO:tasks.rados.rados.0.smithi079.stdout:877: write (excl) oid 48 current snap is 98 2024-04-24T01:58:50.303 INFO:tasks.rados.rados.0.smithi079.stdout:877: seq_num 431 ranges {443817=548087,1788092=438139,3019898=223919} 2024-04-24T01:58:50.311 INFO:tasks.rados.rados.0.smithi079.stdout:877: writing smithi07929762-48 from 443817 to 991904 tid 1 2024-04-24T01:58:50.320 INFO:tasks.rados.rados.0.smithi079.stdout:877: writing smithi07929762-48 from 1788092 to 2226231 tid 2 2024-04-24T01:58:50.327 INFO:tasks.rados.rados.0.smithi079.stdout:877: writing smithi07929762-48 from 3019898 to 3243817 tid 3 2024-04-24T01:58:50.327 INFO:tasks.rados.rados.0.smithi079.stdout:878: snap_create 2024-04-24T01:58:50.331 INFO:tasks.rados.rados.0.smithi079.stdout:871: finishing write tid 2 to smithi07929762-9 2024-04-24T01:58:50.331 INFO:tasks.rados.rados.0.smithi079.stdout:871: finishing write tid 3 to smithi07929762-9 2024-04-24T01:58:50.334 INFO:tasks.rados.rados.0.smithi079.stdout:871: finishing write tid 4 to smithi07929762-9 2024-04-24T01:58:50.334 INFO:tasks.rados.rados.0.smithi079.stdout:871: oid 9 updating version 0 to 287 2024-04-24T01:58:50.334 INFO:tasks.rados.rados.0.smithi079.stdout:871: oid 9 version 287 is already newer than 285 2024-04-24T01:58:50.334 INFO:tasks.rados.rados.0.smithi079.stdout:871: oid 9 version 287 is already newer than 287 2024-04-24T01:58:50.335 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 9 v 287 (ObjNum 426 snap 98 seq_num 426) dirty exists 2024-04-24T01:58:50.335 INFO:tasks.rados.rados.0.smithi079.stdout:871: left oid 9 (ObjNum 426 snap 98 seq_num 426) 2024-04-24T01:58:50.348 INFO:tasks.rados.rados.0.smithi079.stdout:870: finishing write tid 1 to smithi07929762-34 2024-04-24T01:58:50.369 INFO:tasks.rados.rados.0.smithi079.stdout:867: expect (ObjNum 407 snap 96 seq_num 407) 2024-04-24T01:58:50.499 INFO:tasks.rados.rados.0.smithi079.stdout:875: finishing write tid 1 to smithi07929762-28 2024-04-24T01:58:50.499 INFO:tasks.rados.rados.0.smithi079.stdout:870: finishing write tid 2 to smithi07929762-34 2024-04-24T01:58:50.499 INFO:tasks.rados.rados.0.smithi079.stdout:870: finishing write tid 3 to smithi07929762-34 2024-04-24T01:58:50.500 INFO:tasks.rados.rados.0.smithi079.stdout:870: finishing write tid 4 to smithi07929762-34 2024-04-24T01:58:50.500 INFO:tasks.rados.rados.0.smithi079.stdout:874: finishing write tid 1 to smithi07929762-42 2024-04-24T01:58:50.500 INFO:tasks.rados.rados.0.smithi079.stdout:875: finishing write tid 2 to smithi07929762-28 2024-04-24T01:58:50.500 INFO:tasks.rados.rados.0.smithi079.stdout:870: finishing write tid 5 to smithi07929762-34 2024-04-24T01:58:50.500 INFO:tasks.rados.rados.0.smithi079.stdout:870: oid 34 updating version 0 to 223 2024-04-24T01:58:50.500 INFO:tasks.rados.rados.0.smithi079.stdout:870: oid 34 version 223 is already newer than 223 2024-04-24T01:58:50.500 INFO:tasks.rados.rados.0.smithi079.stdout:870: oid 34 version 223 is already newer than 223 2024-04-24T01:58:50.500 INFO:tasks.rados.rados.0.smithi079.stdout:870: oid 34 version 223 is already newer than 223 2024-04-24T01:58:50.500 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 34 v 223 (ObjNum 425 snap 98 seq_num 425) dirty exists 2024-04-24T01:58:50.500 INFO:tasks.rados.rados.0.smithi079.stdout:870: left oid 34 (ObjNum 425 snap 98 seq_num 425) 2024-04-24T01:58:50.500 INFO:tasks.rados.rados.0.smithi079.stdout:875: finishing write tid 3 to smithi07929762-28 2024-04-24T01:58:50.500 INFO:tasks.rados.rados.0.smithi079.stdout:875: finishing write tid 4 to smithi07929762-28 2024-04-24T01:58:50.500 INFO:tasks.rados.rados.0.smithi079.stdout:875: finishing write tid 5 to smithi07929762-28 2024-04-24T01:58:50.501 INFO:tasks.rados.rados.0.smithi079.stdout:875: oid 28 updating version 0 to 258 2024-04-24T01:58:50.501 INFO:tasks.rados.rados.0.smithi079.stdout:875: oid 28 version 258 is already newer than 258 2024-04-24T01:58:50.501 INFO:tasks.rados.rados.0.smithi079.stdout:875: oid 28 version 258 is already newer than 258 2024-04-24T01:58:50.501 INFO:tasks.rados.rados.0.smithi079.stdout:875: oid 28 version 258 is already newer than 258 2024-04-24T01:58:50.501 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 28 v 258 (ObjNum 430 snap 98 seq_num 430) dirty exists 2024-04-24T01:58:50.501 INFO:tasks.rados.rados.0.smithi079.stdout:875: left oid 28 (ObjNum 430 snap 98 seq_num 430) 2024-04-24T01:58:50.501 INFO:tasks.rados.rados.0.smithi079.stdout:877: finishing write tid 1 to smithi07929762-48 2024-04-24T01:58:50.501 INFO:tasks.rados.rados.0.smithi079.stdout:873: finishing write tid 1 to smithi07929762-20 2024-04-24T01:58:50.501 INFO:tasks.rados.rados.0.smithi079.stdout:874: finishing write tid 2 to smithi07929762-42 2024-04-24T01:58:50.501 INFO:tasks.rados.rados.0.smithi079.stdout:874: finishing write tid 3 to smithi07929762-42 2024-04-24T01:58:50.508 INFO:tasks.rados.rados.0.smithi079.stdout:874: finishing write tid 4 to smithi07929762-42 2024-04-24T01:58:50.509 INFO:tasks.rados.rados.0.smithi079.stdout:874: finishing write tid 5 to smithi07929762-42 2024-04-24T01:58:50.509 INFO:tasks.rados.rados.0.smithi079.stdout:874: oid 42 updating version 0 to 212 2024-04-24T01:58:50.509 INFO:tasks.rados.rados.0.smithi079.stdout:874: oid 42 version 212 is already newer than 209 2024-04-24T01:58:50.510 INFO:tasks.rados.rados.0.smithi079.stdout:874: oid 42 version 212 is already newer than 212 2024-04-24T01:58:50.510 INFO:tasks.rados.rados.0.smithi079.stdout:874: oid 42 version 212 is already newer than 212 2024-04-24T01:58:50.510 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 42 v 212 (ObjNum 429 snap 98 seq_num 429) dirty exists 2024-04-24T01:58:50.510 INFO:tasks.rados.rados.0.smithi079.stdout:874: left oid 42 (ObjNum 429 snap 98 seq_num 429) 2024-04-24T01:58:50.510 INFO:tasks.rados.rados.0.smithi079.stdout:872: finishing write tid 1 to smithi07929762-50 2024-04-24T01:58:50.520 INFO:tasks.rados.rados.0.smithi079.stdout:872: finishing write tid 2 to smithi07929762-50 2024-04-24T01:58:50.528 INFO:tasks.rados.rados.0.smithi079.stdout:872: finishing write tid 3 to smithi07929762-50 2024-04-24T01:58:50.533 INFO:tasks.rados.rados.0.smithi079.stdout:872: finishing write tid 4 to smithi07929762-50 2024-04-24T01:58:50.534 INFO:tasks.rados.rados.0.smithi079.stdout:872: finishing write tid 5 to smithi07929762-50 2024-04-24T01:58:50.534 INFO:tasks.rados.rados.0.smithi079.stdout:872: oid 50 updating version 0 to 228 2024-04-24T01:58:50.534 INFO:tasks.rados.rados.0.smithi079.stdout:872: oid 50 version 228 is already newer than 228 2024-04-24T01:58:50.534 INFO:tasks.rados.rados.0.smithi079.stdout:872: oid 50 version 228 is already newer than 228 2024-04-24T01:58:50.535 INFO:tasks.rados.rados.0.smithi079.stdout:872: oid 50 version 228 is already newer than 228 2024-04-24T01:58:50.535 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 50 v 228 (ObjNum 427 snap 98 seq_num 427) dirty exists 2024-04-24T01:58:50.535 INFO:tasks.rados.rados.0.smithi079.stdout:872: left oid 50 (ObjNum 427 snap 98 seq_num 427) 2024-04-24T01:58:50.539 INFO:tasks.rados.rados.0.smithi079.stdout:877: finishing write tid 2 to smithi07929762-48 2024-04-24T01:58:50.551 INFO:tasks.rados.rados.0.smithi079.stdout:877: finishing write tid 3 to smithi07929762-48 2024-04-24T01:58:50.559 INFO:tasks.rados.rados.0.smithi079.stdout:873: finishing write tid 2 to smithi07929762-20 2024-04-24T01:58:50.560 INFO:tasks.rados.rados.0.smithi079.stdout:877: finishing write tid 4 to smithi07929762-48 2024-04-24T01:58:50.561 INFO:tasks.rados.rados.0.smithi079.stdout:877: finishing write tid 5 to smithi07929762-48 2024-04-24T01:58:50.562 INFO:tasks.rados.rados.0.smithi079.stdout:877: oid 48 updating version 0 to 142 2024-04-24T01:58:50.562 INFO:tasks.rados.rados.0.smithi079.stdout:877: oid 48 updating version 142 to 145 2024-04-24T01:58:50.562 INFO:tasks.rados.rados.0.smithi079.stdout:877: oid 48 version 145 is already newer than 145 2024-04-24T01:58:50.562 INFO:tasks.rados.rados.0.smithi079.stdout:877: oid 48 version 145 is already newer than 145 2024-04-24T01:58:50.562 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 48 v 145 (ObjNum 431 snap 98 seq_num 431) dirty exists 2024-04-24T01:58:50.562 INFO:tasks.rados.rados.0.smithi079.stdout:877: left oid 48 (ObjNum 431 snap 98 seq_num 431) 2024-04-24T01:58:50.564 INFO:tasks.rados.rados.0.smithi079.stdout:873: finishing write tid 3 to smithi07929762-20 2024-04-24T01:58:50.566 INFO:tasks.rados.rados.0.smithi079.stdout:873: finishing write tid 4 to smithi07929762-20 2024-04-24T01:58:50.566 INFO:tasks.rados.rados.0.smithi079.stdout:873: oid 20 updating version 0 to 289 2024-04-24T01:58:50.566 INFO:tasks.rados.rados.0.smithi079.stdout:873: oid 20 updating version 289 to 291 2024-04-24T01:58:50.566 INFO:tasks.rados.rados.0.smithi079.stdout:873: oid 20 version 291 is already newer than 291 2024-04-24T01:58:50.566 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 20 v 291 (ObjNum 428 snap 98 seq_num 428) dirty exists 2024-04-24T01:58:50.566 INFO:tasks.rados.rados.0.smithi079.stdout:873: left oid 20 (ObjNum 428 snap 98 seq_num 428) 2024-04-24T01:58:50.979 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T01:58:51.084 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:58:51.082+0000 7f3015f1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:51.088 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:58:51.087+0000 7f3015f1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:51.123 INFO:tasks.rados.rados.0.smithi079.stdout:867: done (9 left) 2024-04-24T01:58:51.123 INFO:tasks.rados.rados.0.smithi079.stdout:870: done (8 left) 2024-04-24T01:58:51.123 INFO:tasks.rados.rados.0.smithi079.stdout:871: done (7 left) 2024-04-24T01:58:51.123 INFO:tasks.rados.rados.0.smithi079.stdout:872: done (6 left) 2024-04-24T01:58:51.123 INFO:tasks.rados.rados.0.smithi079.stdout:873: done (5 left) 2024-04-24T01:58:51.123 INFO:tasks.rados.rados.0.smithi079.stdout:874: done (4 left) 2024-04-24T01:58:51.123 INFO:tasks.rados.rados.0.smithi079.stdout:875: done (3 left) 2024-04-24T01:58:51.123 INFO:tasks.rados.rados.0.smithi079.stdout:876: done (2 left) 2024-04-24T01:58:51.123 INFO:tasks.rados.rados.0.smithi079.stdout:877: done (1 left) 2024-04-24T01:58:51.124 INFO:tasks.rados.rados.0.smithi079.stdout:878: done (0 left) 2024-04-24T01:58:51.124 INFO:tasks.rados.rados.0.smithi079.stdout:879: write (excl) oid 48 current snap is 99 2024-04-24T01:58:51.124 INFO:tasks.rados.rados.0.smithi079.stdout:879: seq_num 432 ranges {460624=623336,1846574=698150,3163674=96950} 2024-04-24T01:58:51.129 INFO:tasks.rados.rados.0.smithi079.stdout:879: writing smithi07929762-48 from 460624 to 1083960 tid 1 2024-04-24T01:58:51.139 INFO:tasks.rados.rados.0.smithi079.stdout:879: writing smithi07929762-48 from 1846574 to 2544724 tid 2 2024-04-24T01:58:51.143 INFO:tasks.rados.rados.0.smithi079.stdout:879: writing smithi07929762-48 from 3163674 to 3260624 tid 3 2024-04-24T01:58:51.143 INFO:tasks.rados.rados.0.smithi079.stdout:880: delete oid 24 current snap is 99 2024-04-24T01:58:51.209 INFO:tasks.rados.rados.0.smithi079.stdout:880: done (1 left) 2024-04-24T01:58:51.209 INFO:tasks.rados.rados.0.smithi079.stdout:881: snap_remove snap 98 2024-04-24T01:58:51.623 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-04-24T01:58:51.623 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-04-24T01:58:51.623 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-04-24T01:58:51.623 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-04-24T01:58:51.732 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:58:51.726+0000 7f44c846d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:51.735 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:58:51.735+0000 7f44c846d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:51.750 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T01:58:51.750 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T01:58:51.750 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T01:58:51.750 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T01:58:51.750 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T01:58:51.750 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T01:58:51.750 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T01:58:51.750 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T01:58:51.750 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T01:58:51.750 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T01:58:51.751 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T01:58:51.751 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T01:58:51.753 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T01:58:51.753 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T01:58:51.753 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-04-24T01:58:51.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:58:51.855+0000 7f88c4b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:51.864 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:58:51.863+0000 7f88c4b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:51.878 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T01:58:51.878 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T01:58:51.879 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T01:58:51.879 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T01:58:51.879 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T01:58:51.879 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T01:58:51.879 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T01:58:51.879 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T01:58:51.879 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T01:58:51.879 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T01:58:51.879 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T01:58:51.879 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T01:58:51.882 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T01:58:51.882 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T01:58:51.882 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-04-24T01:58:51.990 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:58:51.984+0000 7f14da02d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:51.994 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:58:51.993+0000 7f14da02d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:52.010 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T01:58:52.010 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T01:58:52.010 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T01:58:52.010 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T01:58:52.010 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T01:58:52.010 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T01:58:52.010 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T01:58:52.010 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T01:58:52.010 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T01:58:52.010 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T01:58:52.010 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T01:58:52.010 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T01:58:52.013 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T01:58:52.013 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T01:58:52.014 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-04-24T01:58:52.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:58:52.116+0000 7f83a2ead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:52.125 INFO:tasks.rados.rados.0.smithi079.stdout:881: done (1 left) 2024-04-24T01:58:52.125 INFO:tasks.rados.rados.0.smithi079.stdout:882: write (excl) oid 42 current snap is 99 2024-04-24T01:58:52.125 INFO:tasks.rados.rados.0.smithi079.stdout:882: seq_num 434 ranges {494238=490187,1563538=701819,2818520=475718} 2024-04-24T01:58:52.126 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:58:52.126+0000 7f83a2ead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:52.132 INFO:tasks.rados.rados.0.smithi079.stdout:882: writing smithi07929762-42 from 494238 to 984425 tid 1 2024-04-24T01:58:52.140 INFO:tasks.rados.rados.0.smithi079.stdout:882: writing smithi07929762-42 from 1563538 to 2265357 tid 2 2024-04-24T01:58:52.146 INFO:tasks.rados.rados.0.smithi079.stdout:882: writing smithi07929762-42 from 2818520 to 3294238 tid 3 2024-04-24T01:58:52.146 INFO:tasks.rados.rados.0.smithi079.stdout:883: snap_remove snap 97 2024-04-24T01:58:52.150 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T01:58:52.151 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T01:58:52.151 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T01:58:52.151 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T01:58:52.151 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T01:58:52.151 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T01:58:52.151 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T01:58:52.152 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T01:58:52.152 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T01:58:52.152 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T01:58:52.152 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T01:58:52.152 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T01:58:52.156 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T01:58:52.156 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T01:58:53.129 INFO:tasks.rados.rados.0.smithi079.stdout:883: done (2 left) 2024-04-24T01:58:53.129 INFO:tasks.rados.rados.0.smithi079.stdout:884: write (excl) oid 22 current snap is 99 2024-04-24T01:58:53.129 INFO:tasks.rados.rados.0.smithi079.stdout:884: seq_num 435 ranges {511045=565436,1622020=445477,2729590=556855} 2024-04-24T01:58:53.135 INFO:tasks.rados.rados.0.smithi079.stdout:884: writing smithi07929762-22 from 511045 to 1076481 tid 1 2024-04-24T01:58:53.141 INFO:tasks.rados.rados.0.smithi079.stdout:884: writing smithi07929762-22 from 1622020 to 2067497 tid 2 2024-04-24T01:58:53.149 INFO:tasks.rados.rados.0.smithi079.stdout:884: writing smithi07929762-22 from 2729590 to 3286445 tid 3 2024-04-24T01:58:53.149 INFO:tasks.rados.rados.0.smithi079.stdout:885: delete oid 16 current snap is 99 2024-04-24T01:58:53.149 INFO:tasks.rados.rados.0.smithi079.stdout:879: finishing write tid 1 to smithi07929762-48 2024-04-24T01:58:53.149 INFO:tasks.rados.rados.0.smithi079.stdout:879: finishing write tid 2 to smithi07929762-48 2024-04-24T01:58:53.149 INFO:tasks.rados.rados.0.smithi079.stdout:879: finishing write tid 3 to smithi07929762-48 2024-04-24T01:58:53.149 INFO:tasks.rados.rados.0.smithi079.stdout:879: finishing write tid 4 to smithi07929762-48 2024-04-24T01:58:53.149 INFO:tasks.rados.rados.0.smithi079.stdout:879: finishing write tid 5 to smithi07929762-48 2024-04-24T01:58:53.150 INFO:tasks.rados.rados.0.smithi079.stdout:879: oid 48 updating version 0 to 147 2024-04-24T01:58:53.150 INFO:tasks.rados.rados.0.smithi079.stdout:879: oid 48 updating version 147 to 150 2024-04-24T01:58:53.150 INFO:tasks.rados.rados.0.smithi079.stdout:879: oid 48 version 150 is already newer than 150 2024-04-24T01:58:53.150 INFO:tasks.rados.rados.0.smithi079.stdout:879: oid 48 version 150 is already newer than 150 2024-04-24T01:58:53.150 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 48 v 150 (ObjNum 432 snap 99 seq_num 432) dirty exists 2024-04-24T01:58:53.150 INFO:tasks.rados.rados.0.smithi079.stdout:879: left oid 48 (ObjNum 432 snap 99 seq_num 432) 2024-04-24T01:58:53.150 INFO:tasks.rados.rados.0.smithi079.stdout:882: finishing write tid 1 to smithi07929762-42 2024-04-24T01:58:53.150 INFO:tasks.rados.rados.0.smithi079.stdout:882: finishing write tid 2 to smithi07929762-42 2024-04-24T01:58:53.150 INFO:tasks.rados.rados.0.smithi079.stdout:882: finishing write tid 3 to smithi07929762-42 2024-04-24T01:58:53.150 INFO:tasks.rados.rados.0.smithi079.stdout:882: finishing write tid 4 to smithi07929762-42 2024-04-24T01:58:53.150 INFO:tasks.rados.rados.0.smithi079.stdout:882: finishing write tid 5 to smithi07929762-42 2024-04-24T01:58:53.150 INFO:tasks.rados.rados.0.smithi079.stdout:882: oid 42 updating version 0 to 216 2024-04-24T01:58:53.150 INFO:tasks.rados.rados.0.smithi079.stdout:882: oid 42 version 216 is already newer than 213 2024-04-24T01:58:53.151 INFO:tasks.rados.rados.0.smithi079.stdout:882: oid 42 version 216 is already newer than 216 2024-04-24T01:58:53.151 INFO:tasks.rados.rados.0.smithi079.stdout:882: oid 42 version 216 is already newer than 216 2024-04-24T01:58:53.151 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 42 v 216 (ObjNum 434 snap 99 seq_num 434) dirty exists 2024-04-24T01:58:53.151 INFO:tasks.rados.rados.0.smithi079.stdout:882: left oid 42 (ObjNum 434 snap 99 seq_num 434) 2024-04-24T01:58:53.188 INFO:tasks.rados.rados.0.smithi079.stdout:879: done (3 left) 2024-04-24T01:58:53.188 INFO:tasks.rados.rados.0.smithi079.stdout:882: done (2 left) 2024-04-24T01:58:53.188 INFO:tasks.rados.rados.0.smithi079.stdout:885: done (1 left) 2024-04-24T01:58:53.188 INFO:tasks.rados.rados.0.smithi079.stdout:886: write (excl) oid 15 current snap is 99 2024-04-24T01:58:53.188 INFO:tasks.rados.rados.0.smithi079.stdout:886: seq_num 437 ranges {544659=715934,1738984=449146,2784436=560223} 2024-04-24T01:58:53.195 INFO:tasks.rados.rados.0.smithi079.stdout:886: writing smithi07929762-15 from 544659 to 1260593 tid 1 2024-04-24T01:58:53.200 INFO:tasks.rados.rados.0.smithi079.stdout:886: writing smithi07929762-15 from 1738984 to 2188130 tid 2 2024-04-24T01:58:53.207 INFO:tasks.rados.rados.0.smithi079.stdout:886: writing smithi07929762-15 from 2784436 to 3344659 tid 3 2024-04-24T01:58:53.207 INFO:tasks.rados.rados.0.smithi079.stdout:887: write oid 33 current snap is 99 2024-04-24T01:58:53.207 INFO:tasks.rados.rados.0.smithi079.stdout:887: seq_num 438 ranges {561466=791183,2081113=592804,3095506=265960} 2024-04-24T01:58:53.214 INFO:tasks.rados.rados.0.smithi079.stdout:887: writing smithi07929762-33 from 561466 to 1352649 tid 1 2024-04-24T01:58:53.222 INFO:tasks.rados.rados.0.smithi079.stdout:887: writing smithi07929762-33 from 2081113 to 2673917 tid 2 2024-04-24T01:58:53.226 INFO:tasks.rados.rados.0.smithi079.stdout:887: writing smithi07929762-33 from 3095506 to 3361466 tid 3 2024-04-24T01:58:53.226 INFO:tasks.rados.rados.0.smithi079.stdout:888: delete oid 46 current snap is 99 2024-04-24T01:58:53.244 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T01:58:53.265 INFO:tasks.rados.rados.0.smithi079.stdout:884: finishing write tid 1 to smithi07929762-22 2024-04-24T01:58:53.265 INFO:tasks.rados.rados.0.smithi079.stdout:888: done (3 left) 2024-04-24T01:58:53.265 INFO:tasks.rados.rados.0.smithi079.stdout:889: write oid 48 current snap is 99 2024-04-24T01:58:53.265 INFO:tasks.rados.rados.0.smithi079.stdout:889: seq_num 440 ranges {595080=541681,1798077=596473,3150352=244728} 2024-04-24T01:58:53.274 INFO:tasks.rados.rados.0.smithi079.stdout:889: writing smithi07929762-48 from 595080 to 1136761 tid 1 2024-04-24T01:58:53.285 INFO:tasks.rados.rados.0.smithi079.stdout:889: writing smithi07929762-48 from 1798077 to 2394550 tid 2 2024-04-24T01:58:53.292 INFO:tasks.rados.rados.0.smithi079.stdout:889: writing smithi07929762-48 from 3150352 to 3395080 tid 3 2024-04-24T01:58:53.292 INFO:tasks.rados.rados.0.smithi079.stdout:890: write (excl) oid 21 current snap is 99 2024-04-24T01:58:53.292 INFO:tasks.rados.rados.0.smithi079.stdout:890: seq_num 441 ranges {611887=733283,1972912=456484,2894128=517759} 2024-04-24T01:58:53.303 INFO:tasks.rados.rados.0.smithi079.stdout:890: writing smithi07929762-21 from 611887 to 1345170 tid 1 2024-04-24T01:58:53.310 INFO:tasks.rados.rados.0.smithi079.stdout:890: writing smithi07929762-21 from 1972912 to 2429396 tid 2 2024-04-24T01:58:53.318 INFO:tasks.rados.rados.0.smithi079.stdout:890: writing smithi07929762-21 from 2894128 to 3411887 tid 3 2024-04-24T01:58:53.318 INFO:tasks.rados.rados.0.smithi079.stdout:891: read oid 50 snap -1 2024-04-24T01:58:53.319 INFO:tasks.rados.rados.0.smithi079.stdout:891: expect (ObjNum 427 snap 98 seq_num 427) 2024-04-24T01:58:53.319 INFO:tasks.rados.rados.0.smithi079.stdout:892: read oid 18 snap -1 2024-04-24T01:58:53.319 INFO:tasks.rados.rados.0.smithi079.stdout:892: expect deleted 2024-04-24T01:58:53.319 INFO:tasks.rados.rados.0.smithi079.stdout:893: write oid 2 current snap is 99 2024-04-24T01:58:53.319 INFO:tasks.rados.rados.0.smithi079.stdout:893: seq_num 442 ranges {628694=408532,1515041=600142,2805198=552112} 2024-04-24T01:58:53.325 INFO:tasks.rados.rados.0.smithi079.stdout:893: writing smithi07929762-2 from 628694 to 1037226 tid 1 2024-04-24T01:58:53.334 INFO:tasks.rados.rados.0.smithi079.stdout:893: writing smithi07929762-2 from 1515041 to 2115183 tid 2 2024-04-24T01:58:53.343 INFO:tasks.rados.rados.0.smithi079.stdout:893: writing smithi07929762-2 from 2805198 to 3357310 tid 3 2024-04-24T01:58:53.344 INFO:tasks.rados.rados.0.smithi079.stdout:894: read oid 47 snap -1 2024-04-24T01:58:53.344 INFO:tasks.rados.rados.0.smithi079.stdout:894: expect deleted 2024-04-24T01:58:53.344 INFO:tasks.rados.rados.0.smithi079.stdout:895: snap_create 2024-04-24T01:58:53.344 INFO:tasks.rados.rados.0.smithi079.stdout:886: finishing write tid 1 to smithi07929762-15 2024-04-24T01:58:53.378 INFO:tasks.rados.rados.0.smithi079.stdout:887: finishing write tid 1 to smithi07929762-33 2024-04-24T01:58:53.379 INFO:tasks.rados.rados.0.smithi079.stdout:889: finishing write tid 1 to smithi07929762-48 2024-04-24T01:58:53.399 INFO:tasks.rados.rados.0.smithi079.stdout:887: finishing write tid 2 to smithi07929762-33 2024-04-24T01:58:53.407 INFO:tasks.rados.rados.0.smithi079.stdout:884: finishing write tid 2 to smithi07929762-22 2024-04-24T01:58:53.409 INFO:tasks.rados.rados.0.smithi079.stdout:891: expect (ObjNum 427 snap 98 seq_num 427) 2024-04-24T01:58:53.409 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:58:53.406+0000 7f640966d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:53.416 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:58:53.415+0000 7f640966d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:53.666 INFO:tasks.rados.rados.0.smithi079.stdout:889: finishing write tid 2 to smithi07929762-48 2024-04-24T01:58:53.666 INFO:tasks.rados.rados.0.smithi079.stdout:889: finishing write tid 3 to smithi07929762-48 2024-04-24T01:58:53.666 INFO:tasks.rados.rados.0.smithi079.stdout:884: finishing write tid 3 to smithi07929762-22 2024-04-24T01:58:53.666 INFO:tasks.rados.rados.0.smithi079.stdout:893: finishing write tid 1 to smithi07929762-2 2024-04-24T01:58:53.666 INFO:tasks.rados.rados.0.smithi079.stdout:884: finishing write tid 4 to smithi07929762-22 2024-04-24T01:58:53.666 INFO:tasks.rados.rados.0.smithi079.stdout:887: finishing write tid 3 to smithi07929762-33 2024-04-24T01:58:53.666 INFO:tasks.rados.rados.0.smithi079.stdout:884: finishing write tid 5 to smithi07929762-22 2024-04-24T01:58:53.666 INFO:tasks.rados.rados.0.smithi079.stdout:884: oid 22 updating version 0 to 138 2024-04-24T01:58:53.666 INFO:tasks.rados.rados.0.smithi079.stdout:884: oid 22 version 138 is already newer than 138 2024-04-24T01:58:53.666 INFO:tasks.rados.rados.0.smithi079.stdout:884: oid 22 version 138 is already newer than 135 2024-04-24T01:58:53.666 INFO:tasks.rados.rados.0.smithi079.stdout:884: oid 22 version 138 is already newer than 138 2024-04-24T01:58:53.666 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 22 v 138 (ObjNum 435 snap 99 seq_num 435) dirty exists 2024-04-24T01:58:53.667 INFO:tasks.rados.rados.0.smithi079.stdout:884: left oid 22 (ObjNum 435 snap 99 seq_num 435) 2024-04-24T01:58:53.667 INFO:tasks.rados.rados.0.smithi079.stdout:887: finishing write tid 4 to smithi07929762-33 2024-04-24T01:58:53.667 INFO:tasks.rados.rados.0.smithi079.stdout:887: finishing write tid 5 to smithi07929762-33 2024-04-24T01:58:53.667 INFO:tasks.rados.rados.0.smithi079.stdout:887: oid 33 updating version 0 to 299 2024-04-24T01:58:53.667 INFO:tasks.rados.rados.0.smithi079.stdout:887: oid 33 version 299 is already newer than 299 2024-04-24T01:58:53.667 INFO:tasks.rados.rados.0.smithi079.stdout:887: oid 33 version 299 is already newer than 299 2024-04-24T01:58:53.667 INFO:tasks.rados.rados.0.smithi079.stdout:887: oid 33 version 299 is already newer than 299 2024-04-24T01:58:53.667 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 33 v 299 (ObjNum 438 snap 99 seq_num 438) dirty exists 2024-04-24T01:58:53.667 INFO:tasks.rados.rados.0.smithi079.stdout:887: left oid 33 (ObjNum 438 snap 99 seq_num 438) 2024-04-24T01:58:53.667 INFO:tasks.rados.rados.0.smithi079.stdout:889: finishing write tid 4 to smithi07929762-48 2024-04-24T01:58:53.667 INFO:tasks.rados.rados.0.smithi079.stdout:889: finishing write tid 5 to smithi07929762-48 2024-04-24T01:58:53.667 INFO:tasks.rados.rados.0.smithi079.stdout:889: oid 48 updating version 0 to 156 2024-04-24T01:58:53.667 INFO:tasks.rados.rados.0.smithi079.stdout:889: oid 48 version 156 is already newer than 156 2024-04-24T01:58:53.667 INFO:tasks.rados.rados.0.smithi079.stdout:889: oid 48 version 156 is already newer than 156 2024-04-24T01:58:53.668 INFO:tasks.rados.rados.0.smithi079.stdout:889: oid 48 version 156 is already newer than 156 2024-04-24T01:58:53.668 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 48 v 156 (ObjNum 440 snap 99 seq_num 440) dirty exists 2024-04-24T01:58:53.668 INFO:tasks.rados.rados.0.smithi079.stdout:889: left oid 48 (ObjNum 440 snap 99 seq_num 440) 2024-04-24T01:58:53.668 INFO:tasks.rados.rados.0.smithi079.stdout:890: finishing write tid 1 to smithi07929762-21 2024-04-24T01:58:53.668 INFO:tasks.rados.rados.0.smithi079.stdout:886: finishing write tid 2 to smithi07929762-15 2024-04-24T01:58:53.668 INFO:tasks.rados.rados.0.smithi079.stdout:893: finishing write tid 2 to smithi07929762-2 2024-04-24T01:58:53.668 INFO:tasks.rados.rados.0.smithi079.stdout:886: finishing write tid 3 to smithi07929762-15 2024-04-24T01:58:53.668 INFO:tasks.rados.rados.0.smithi079.stdout:893: finishing write tid 3 to smithi07929762-2 2024-04-24T01:58:53.668 INFO:tasks.rados.rados.0.smithi079.stdout:886: finishing write tid 4 to smithi07929762-15 2024-04-24T01:58:53.668 INFO:tasks.rados.rados.0.smithi079.stdout:886: finishing write tid 5 to smithi07929762-15 2024-04-24T01:58:53.668 INFO:tasks.rados.rados.0.smithi079.stdout:886: oid 15 updating version 0 to 227 2024-04-24T01:58:53.668 INFO:tasks.rados.rados.0.smithi079.stdout:886: oid 15 version 227 is already newer than 224 2024-04-24T01:58:53.668 INFO:tasks.rados.rados.0.smithi079.stdout:886: oid 15 version 227 is already newer than 227 2024-04-24T01:58:53.668 INFO:tasks.rados.rados.0.smithi079.stdout:886: oid 15 version 227 is already newer than 227 2024-04-24T01:58:53.668 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 15 v 227 (ObjNum 437 snap 99 seq_num 437) dirty exists 2024-04-24T01:58:53.669 INFO:tasks.rados.rados.0.smithi079.stdout:886: left oid 15 (ObjNum 437 snap 99 seq_num 437) 2024-04-24T01:58:53.669 INFO:tasks.rados.rados.0.smithi079.stdout:890: finishing write tid 2 to smithi07929762-21 2024-04-24T01:58:53.669 INFO:tasks.rados.rados.0.smithi079.stdout:893: finishing write tid 4 to smithi07929762-2 2024-04-24T01:58:53.669 INFO:tasks.rados.rados.0.smithi079.stdout:893: finishing write tid 5 to smithi07929762-2 2024-04-24T01:58:53.669 INFO:tasks.rados.rados.0.smithi079.stdout:893: oid 2 updating version 0 to 58 2024-04-24T01:58:53.669 INFO:tasks.rados.rados.0.smithi079.stdout:893: oid 2 version 58 is already newer than 58 2024-04-24T01:58:53.669 INFO:tasks.rados.rados.0.smithi079.stdout:893: oid 2 version 58 is already newer than 58 2024-04-24T01:58:53.669 INFO:tasks.rados.rados.0.smithi079.stdout:893: oid 2 version 58 is already newer than 58 2024-04-24T01:58:53.669 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 2 v 58 (ObjNum 442 snap 99 seq_num 442) dirty exists 2024-04-24T01:58:53.669 INFO:tasks.rados.rados.0.smithi079.stdout:893: left oid 2 (ObjNum 442 snap 99 seq_num 442) 2024-04-24T01:58:53.669 INFO:tasks.rados.rados.0.smithi079.stdout:890: finishing write tid 3 to smithi07929762-21 2024-04-24T01:58:53.669 INFO:tasks.rados.rados.0.smithi079.stdout:890: finishing write tid 4 to smithi07929762-21 2024-04-24T01:58:53.669 INFO:tasks.rados.rados.0.smithi079.stdout:890: finishing write tid 5 to smithi07929762-21 2024-04-24T01:58:53.669 INFO:tasks.rados.rados.0.smithi079.stdout:890: oid 21 updating version 0 to 262 2024-04-24T01:58:53.669 INFO:tasks.rados.rados.0.smithi079.stdout:890: oid 21 version 262 is already newer than 259 2024-04-24T01:58:53.670 INFO:tasks.rados.rados.0.smithi079.stdout:890: oid 21 version 262 is already newer than 262 2024-04-24T01:58:53.670 INFO:tasks.rados.rados.0.smithi079.stdout:890: oid 21 version 262 is already newer than 262 2024-04-24T01:58:53.670 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 21 v 262 (ObjNum 441 snap 99 seq_num 441) dirty exists 2024-04-24T01:58:53.670 INFO:tasks.rados.rados.0.smithi079.stdout:890: left oid 21 (ObjNum 441 snap 99 seq_num 441) 2024-04-24T01:58:54.129 INFO:tasks.rados.rados.0.smithi079.stdout:884: done (9 left) 2024-04-24T01:58:54.130 INFO:tasks.rados.rados.0.smithi079.stdout:886: done (8 left) 2024-04-24T01:58:54.130 INFO:tasks.rados.rados.0.smithi079.stdout:887: done (7 left) 2024-04-24T01:58:54.130 INFO:tasks.rados.rados.0.smithi079.stdout:889: done (6 left) 2024-04-24T01:58:54.130 INFO:tasks.rados.rados.0.smithi079.stdout:890: done (5 left) 2024-04-24T01:58:54.130 INFO:tasks.rados.rados.0.smithi079.stdout:891: done (4 left) 2024-04-24T01:58:54.130 INFO:tasks.rados.rados.0.smithi079.stdout:892: done (3 left) 2024-04-24T01:58:54.130 INFO:tasks.rados.rados.0.smithi079.stdout:893: done (2 left) 2024-04-24T01:58:54.130 INFO:tasks.rados.rados.0.smithi079.stdout:894: done (1 left) 2024-04-24T01:58:54.130 INFO:tasks.rados.rados.0.smithi079.stdout:895: done (0 left) 2024-04-24T01:58:54.130 INFO:tasks.rados.rados.0.smithi079.stdout:896: snap_create 2024-04-24T01:58:55.136 INFO:tasks.rados.rados.0.smithi079.stdout:896: done (0 left) 2024-04-24T01:58:55.136 INFO:tasks.rados.rados.0.smithi079.stdout:897: delete oid 13 current snap is 101 2024-04-24T01:58:55.187 INFO:tasks.rados.rados.0.smithi079.stdout:897: done (0 left) 2024-04-24T01:58:55.187 INFO:tasks.rados.rados.0.smithi079.stdout:898: delete oid 45 current snap is 101 2024-04-24T01:58:55.225 INFO:tasks.rados.rados.0.smithi079.stdout:898: done (0 left) 2024-04-24T01:58:55.226 INFO:tasks.rados.rados.0.smithi079.stdout:899: delete oid 48 current snap is 101 2024-04-24T01:58:55.257 INFO:tasks.rados.rados.0.smithi079.stdout:899: done (0 left) 2024-04-24T01:58:55.258 INFO:tasks.rados.rados.0.smithi079.stdout:900: read oid 11 snap -1 2024-04-24T01:58:55.258 INFO:tasks.rados.rados.0.smithi079.stdout:900: expect (ObjNum 396 snap 94 seq_num 396) 2024-04-24T01:58:55.258 INFO:tasks.rados.rados.0.smithi079.stdout:901: read oid 46 snap -1 2024-04-24T01:58:55.258 INFO:tasks.rados.rados.0.smithi079.stdout:901: expect deleted 2024-04-24T01:58:55.258 INFO:tasks.rados.rados.0.smithi079.stdout:902: read oid 16 snap -1 2024-04-24T01:58:55.258 INFO:tasks.rados.rados.0.smithi079.stdout:902: expect deleted 2024-04-24T01:58:55.258 INFO:tasks.rados.rados.0.smithi079.stdout:903: snap_create 2024-04-24T01:58:55.274 INFO:tasks.rados.rados.0.smithi079.stdout:900: expect (ObjNum 396 snap 94 seq_num 396) 2024-04-24T01:58:55.529 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T01:58:55.633 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:58:55.631+0000 7fb96635d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:55.637 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:58:55.636+0000 7fb96635d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:56.043 INFO:tasks.rados.rados.0.smithi079.stdout:900: done (3 left) 2024-04-24T01:58:56.043 INFO:tasks.rados.rados.0.smithi079.stdout:901: done (2 left) 2024-04-24T01:58:56.043 INFO:tasks.rados.rados.0.smithi079.stdout:902: done (1 left) 2024-04-24T01:58:56.043 INFO:tasks.rados.rados.0.smithi079.stdout:903: done (0 left) 2024-04-24T01:58:56.043 INFO:tasks.rados.rados.0.smithi079.stdout:904: delete oid 49 current snap is 102 2024-04-24T01:58:56.091 INFO:tasks.rados.rados.0.smithi079.stdout:904: done (0 left) 2024-04-24T01:58:56.091 INFO:tasks.rados.rados.0.smithi079.stdout:905: write oid 8 current snap is 102 2024-04-24T01:58:56.091 INFO:tasks.rados.rados.0.smithi079.stdout:905: seq_num 447 ranges {712729=784777,2091098=467491,3342313=170416} 2024-04-24T01:58:56.099 INFO:tasks.rados.rados.0.smithi079.stdout:905: writing smithi07929762-8 from 712729 to 1497506 tid 1 2024-04-24T01:58:56.105 INFO:tasks.rados.rados.0.smithi079.stdout:905: writing smithi07929762-8 from 2091098 to 2558589 tid 2 2024-04-24T01:58:56.110 INFO:tasks.rados.rados.0.smithi079.stdout:905: writing smithi07929762-8 from 3342313 to 3512729 tid 3 2024-04-24T01:58:56.110 INFO:tasks.rados.rados.0.smithi079.stdout:906: write (excl) oid 46 current snap is 102 2024-04-24T01:58:56.110 INFO:tasks.rados.rados.0.smithi079.stdout:906: seq_num 448 ranges {729536=460026,1749580=611149,2853383=452450} 2024-04-24T01:58:56.117 INFO:tasks.rados.rados.0.smithi079.stdout:906: writing smithi07929762-46 from 729536 to 1189562 tid 1 2024-04-24T01:58:56.124 INFO:tasks.rados.rados.0.smithi079.stdout:906: writing smithi07929762-46 from 1749580 to 2360729 tid 2 2024-04-24T01:58:56.130 INFO:tasks.rados.rados.0.smithi079.stdout:906: writing smithi07929762-46 from 2853383 to 3305833 tid 3 2024-04-24T01:58:56.130 INFO:tasks.rados.rados.0.smithi079.stdout:907: snap_remove snap 101 2024-04-24T01:58:57.047 INFO:tasks.rados.rados.0.smithi079.stdout:907: done (2 left) 2024-04-24T01:58:57.047 INFO:tasks.rados.rados.0.smithi079.stdout:908: snap_remove snap 100 2024-04-24T01:58:57.157 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-04-24T01:58:57.157 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-04-24T01:58:57.157 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T01:58:57.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:58:57.264+0000 7f42bc02d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:57.272 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:58:57.272+0000 7f42bc02d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:57.459 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:58:57.460 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-24T01:58:57.472 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":324,"stamp":"2024-04-24T01:58:57.044885+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":87702798,"num_objects":34,"num_object_clones":3,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1736,"num_read_kb":1128904,"num_write":1613,"num_write_kb":403325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3196,"ondisk_log_size":3196,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":295432,"kb_used_data":154356,"kb_used_omap":0,"kb_used_meta":132224,"kb_avail":377191928,"statfs":{"total":386547056640,"available":386244534272,"internally_reserved":0,"allocated":158060544,"data_stored":156864236,"data_compressed":89889,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":135397376},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":7142125,"num_objects":-2,"num_object_clones":0,"num_object_copies":-6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":-1,"num_read":60,"num_read_kb":35222,"num_write":87,"num_write_kb":23711,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.917763"},"pg_stats":[{"pgid":"3.f","version":"217'156","reported_seq":799,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-04-24T01:58:54.177992+0000","last_change":"2024-04-24T01:58:53.154263+0000","last_active":"2024-04-24T01:58:54.177992+0000","last_peered":"2024-04-24T01:58:54.177992+0000","last_clean":"2024-04-24T01:58:54.177992+0000","last_became_active":"2024-04-24T01:55:30.386726+0000","last_became_peered":"2024-04-24T01:55:30.386726+0000","last_unstale":"2024-04-24T01:58:54.177992+0000","last_undegraded":"2024-04-24T01:58:54.177992+0000","last_fullsized":"2024-04-24T01:58:54.177992+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":156,"log_dups_size":0,"ondisk_log_size":156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":14,"snaptrim_duration":0,"stat_sum":{"num_bytes":5680898,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":105,"num_read_kb":89470,"num_write":73,"num_write_kb":20366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"217'262","reported_seq":874,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-04-24T01:58:54.143847+0000","last_change":"2024-04-24T01:58:53.252896+0000","last_active":"2024-04-24T01:58:54.143847+0000","last_peered":"2024-04-24T01:58:54.143847+0000","last_clean":"2024-04-24T01:58:54.143847+0000","last_became_active":"2024-04-24T01:55:30.321866+0000","last_became_peered":"2024-04-24T01:55:30.321866+0000","last_unstale":"2024-04-24T01:58:54.143847+0000","last_undegraded":"2024-04-24T01:58:54.143847+0000","last_fullsized":"2024-04-24T01:58:54.143847+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":262,"log_dups_size":0,"ondisk_log_size":262,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":20,"snaptrim_duration":0,"stat_sum":{"num_bytes":13227340,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":152,"num_read_kb":106053,"num_write":126,"num_write_kb":34573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"216'216","reported_seq":818,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-04-24T01:58:52.214548+0000","last_change":"2024-04-24T01:58:52.165796+0000","last_active":"2024-04-24T01:58:52.214548+0000","last_peered":"2024-04-24T01:58:52.214548+0000","last_clean":"2024-04-24T01:58:52.214548+0000","last_became_active":"2024-04-24T01:55:30.320186+0000","last_became_peered":"2024-04-24T01:55:30.320186+0000","last_unstale":"2024-04-24T01:58:52.214548+0000","last_undegraded":"2024-04-24T01:58:52.214548+0000","last_fullsized":"2024-04-24T01:58:52.214548+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":216,"log_dups_size":0,"ondisk_log_size":216,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":21,"snaptrim_duration":0,"stat_sum":{"num_bytes":5949810,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":92,"num_read_kb":52293,"num_write":119,"num_write_kb":33001,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"217'233","reported_seq":859,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-04-24T01:58:54.173389+0000","last_change":"2024-04-24T01:58:53.222404+0000","last_active":"2024-04-24T01:58:54.173389+0000","last_peered":"2024-04-24T01:58:54.173389+0000","last_clean":"2024-04-24T01:58:54.173389+0000","last_became_active":"2024-04-24T01:55:30.288081+0000","last_became_peered":"2024-04-24T01:55:30.288081+0000","last_unstale":"2024-04-24T01:58:54.173389+0000","last_undegraded":"2024-04-24T01:58:54.173389+0000","last_fullsized":"2024-04-24T01:58:54.173389+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":233,"log_dups_size":0,"ondisk_log_size":233,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":23,"snaptrim_duration":0,"stat_sum":{"num_bytes":6319564,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":126,"num_read_kb":92548,"num_write":105,"num_write_kb":28763,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"208'62","reported_seq":691,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-04-24T01:58:53.177070+0000","last_change":"2024-04-24T01:58:53.181982+0000","last_active":"2024-04-24T01:58:53.177070+0000","last_peered":"2024-04-24T01:58:53.177070+0000","last_clean":"2024-04-24T01:58:53.177070+0000","last_became_active":"2024-04-24T01:55:30.333815+0000","last_became_peered":"2024-04-24T01:55:30.333815+0000","last_unstale":"2024-04-24T01:58:53.177070+0000","last_undegraded":"2024-04-24T01:58:53.177070+0000","last_fullsized":"2024-04-24T01:58:53.177070+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":5,"snaptrim_duration":0,"stat_sum":{"num_bytes":3798382,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":32,"num_read_kb":17556,"num_write":36,"num_write_kb":10196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"217'439","reported_seq":1022,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-04-24T01:58:54.172734+0000","last_change":"2024-04-24T01:58:53.155904+0000","last_active":"2024-04-24T01:58:54.172734+0000","last_peered":"2024-04-24T01:58:54.172734+0000","last_clean":"2024-04-24T01:58:54.172734+0000","last_became_active":"2024-04-24T01:55:30.315075+0000","last_became_peered":"2024-04-24T01:55:30.315075+0000","last_unstale":"2024-04-24T01:58:54.172734+0000","last_undegraded":"2024-04-24T01:58:54.172734+0000","last_fullsized":"2024-04-24T01:58:54.172734+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":439,"log_dups_size":0,"ondisk_log_size":439,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":40,"snaptrim_duration":0,"stat_sum":{"num_bytes":8874294,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":196,"num_read_kb":122518,"num_write":190,"num_write_kb":50968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"213'74","reported_seq":711,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-04-24T01:58:53.151436+0000","last_change":"2024-04-24T01:58:53.160042+0000","last_active":"2024-04-24T01:58:53.151436+0000","last_peered":"2024-04-24T01:58:53.151436+0000","last_clean":"2024-04-24T01:58:53.151436+0000","last_became_active":"2024-04-24T01:55:30.350300+0000","last_became_peered":"2024-04-24T01:55:30.350300+0000","last_unstale":"2024-04-24T01:58:53.151436+0000","last_undegraded":"2024-04-24T01:58:53.151436+0000","last_fullsized":"2024-04-24T01:58:53.151436+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":8,"snaptrim_duration":0,"stat_sum":{"num_bytes":3058940,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":16965,"num_write":47,"num_write_kb":11517,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"163'8","reported_seq":629,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-04-24T01:58:52.170652+0000","last_change":"2024-04-24T01:58:52.178743+0000","last_active":"2024-04-24T01:58:52.170652+0000","last_peered":"2024-04-24T01:58:52.170652+0000","last_clean":"2024-04-24T01:58:52.170652+0000","last_became_active":"2024-04-24T01:55:30.376813+0000","last_became_peered":"2024-04-24T01:55:30.376813+0000","last_unstale":"2024-04-24T01:58:52.170652+0000","last_undegraded":"2024-04-24T01:58:52.170652+0000","last_fullsized":"2024-04-24T01:58:52.170652+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'97","reported_seq":675,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-04-24T01:58:53.158096+0000","last_change":"2024-04-24T01:55:16.196470+0000","last_active":"2024-04-24T01:58:53.158096+0000","last_peered":"2024-04-24T01:58:53.158096+0000","last_clean":"2024-04-24T01:58:53.158096+0000","last_became_active":"2024-04-24T01:55:16.190464+0000","last_became_peered":"2024-04-24T01:55:16.190464+0000","last_unstale":"2024-04-24T01:58:53.158096+0000","last_undegraded":"2024-04-24T01:58:53.158096+0000","last_fullsized":"2024-04-24T01:58:53.158096+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:15.127115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:15.127115+0000","last_clean_scrub_stamp":"2024-04-24T01:55:15.127115+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":535,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-04-24T01:58:52.171214+0000","last_change":"2024-04-24T01:55:18.236148+0000","last_active":"2024-04-24T01:58:52.171214+0000","last_peered":"2024-04-24T01:58:52.171214+0000","last_clean":"2024-04-24T01:58:52.171214+0000","last_became_active":"2024-04-24T01:55:18.230154+0000","last_became_peered":"2024-04-24T01:55:18.230154+0000","last_unstale":"2024-04-24T01:58:52.171214+0000","last_undegraded":"2024-04-24T01:58:52.171214+0000","last_fullsized":"2024-04-24T01:58:52.171214+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"213'338","reported_seq":943,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-04-24T01:58:52.173217+0000","last_change":"2024-04-24T01:58:52.178946+0000","last_active":"2024-04-24T01:58:52.173217+0000","last_peered":"2024-04-24T01:58:52.173217+0000","last_clean":"2024-04-24T01:58:52.173217+0000","last_became_active":"2024-04-24T01:55:30.316222+0000","last_became_peered":"2024-04-24T01:55:30.316222+0000","last_unstale":"2024-04-24T01:58:52.173217+0000","last_undegraded":"2024-04-24T01:58:52.173217+0000","last_fullsized":"2024-04-24T01:58:52.173217+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":338,"log_dups_size":0,"ondisk_log_size":338,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":27,"snaptrim_duration":0,"stat_sum":{"num_bytes":5008618,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":172,"num_read_kb":111692,"num_write":170,"num_write_kb":47239,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":553,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-04-24T01:58:54.181484+0000","last_change":"2024-04-24T01:55:18.265905+0000","last_active":"2024-04-24T01:58:54.181484+0000","last_peered":"2024-04-24T01:58:54.181484+0000","last_clean":"2024-04-24T01:58:54.181484+0000","last_became_active":"2024-04-24T01:55:18.257020+0000","last_became_peered":"2024-04-24T01:55:18.257020+0000","last_unstale":"2024-04-24T01:58:54.181484+0000","last_undegraded":"2024-04-24T01:58:54.181484+0000","last_fullsized":"2024-04-24T01:58:54.181484+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"217'274","reported_seq":872,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-04-24T01:58:53.189085+0000","last_change":"2024-04-24T01:58:53.227450+0000","last_active":"2024-04-24T01:58:53.189085+0000","last_peered":"2024-04-24T01:58:53.189085+0000","last_clean":"2024-04-24T01:58:53.189085+0000","last_became_active":"2024-04-24T01:55:30.344129+0000","last_became_peered":"2024-04-24T01:55:30.344129+0000","last_unstale":"2024-04-24T01:58:53.189085+0000","last_undegraded":"2024-04-24T01:58:53.189085+0000","last_fullsized":"2024-04-24T01:58:53.189085+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":274,"log_dups_size":0,"ondisk_log_size":274,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":20,"snaptrim_duration":0,"stat_sum":{"num_bytes":2638765,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":122,"num_read_kb":72533,"num_write":120,"num_write_kb":33534,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":535,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-04-24T01:58:52.157296+0000","last_change":"2024-04-24T01:55:18.215251+0000","last_active":"2024-04-24T01:58:52.157296+0000","last_peered":"2024-04-24T01:58:52.157296+0000","last_clean":"2024-04-24T01:58:52.157296+0000","last_became_active":"2024-04-24T01:55:18.206605+0000","last_became_peered":"2024-04-24T01:55:18.206605+0000","last_unstale":"2024-04-24T01:58:52.157296+0000","last_undegraded":"2024-04-24T01:58:52.157296+0000","last_fullsized":"2024-04-24T01:58:52.157296+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"217'138","reported_seq":775,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-04-24T01:58:53.278438+0000","last_change":"2024-04-24T01:58:53.189084+0000","last_active":"2024-04-24T01:58:53.278438+0000","last_peered":"2024-04-24T01:58:53.278438+0000","last_clean":"2024-04-24T01:58:53.278438+0000","last_became_active":"2024-04-24T01:55:30.401366+0000","last_became_peered":"2024-04-24T01:55:30.401366+0000","last_unstale":"2024-04-24T01:58:53.278438+0000","last_undegraded":"2024-04-24T01:58:53.278438+0000","last_fullsized":"2024-04-24T01:58:53.278438+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":10,"snaptrim_duration":0,"stat_sum":{"num_bytes":5529635,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":110,"num_read_kb":83726,"num_write":51,"num_write_kb":13418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":551,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-04-24T01:58:53.154247+0000","last_change":"2024-04-24T01:55:18.230893+0000","last_active":"2024-04-24T01:58:53.154247+0000","last_peered":"2024-04-24T01:58:53.154247+0000","last_clean":"2024-04-24T01:58:53.154247+0000","last_became_active":"2024-04-24T01:55:18.225188+0000","last_became_peered":"2024-04-24T01:55:18.225188+0000","last_unstale":"2024-04-24T01:58:53.154247+0000","last_undegraded":"2024-04-24T01:58:53.154247+0000","last_fullsized":"2024-04-24T01:58:53.154247+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"208'225","reported_seq":815,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-04-24T01:58:54.144734+0000","last_change":"2024-04-24T01:58:53.246940+0000","last_active":"2024-04-24T01:58:54.144734+0000","last_peered":"2024-04-24T01:58:54.144734+0000","last_clean":"2024-04-24T01:58:54.144734+0000","last_became_active":"2024-04-24T01:55:30.343318+0000","last_became_peered":"2024-04-24T01:55:30.343318+0000","last_unstale":"2024-04-24T01:58:54.144734+0000","last_undegraded":"2024-04-24T01:58:54.144734+0000","last_fullsized":"2024-04-24T01:58:54.144734+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":225,"log_dups_size":0,"ondisk_log_size":225,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":20,"snaptrim_duration":0,"stat_sum":{"num_bytes":2302625,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":98,"num_read_kb":79025,"num_write":97,"num_write_kb":26701,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"214'291","reported_seq":895,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-04-24T01:58:52.172088+0000","last_change":"2024-04-24T01:58:52.178004+0000","last_active":"2024-04-24T01:58:52.172088+0000","last_peered":"2024-04-24T01:58:52.172088+0000","last_clean":"2024-04-24T01:58:52.172088+0000","last_became_active":"2024-04-24T01:55:30.345308+0000","last_became_peered":"2024-04-24T01:55:30.345308+0000","last_unstale":"2024-04-24T01:58:52.172088+0000","last_undegraded":"2024-04-24T01:58:52.172088+0000","last_fullsized":"2024-04-24T01:58:52.172088+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":291,"log_dups_size":0,"ondisk_log_size":291,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":22,"snaptrim_duration":0,"stat_sum":{"num_bytes":10185207,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":161,"num_read_kb":110061,"num_write":129,"num_write_kb":33537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":531,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-04-24T01:58:54.154277+0000","last_change":"2024-04-24T01:55:18.224238+0000","last_active":"2024-04-24T01:58:54.154277+0000","last_peered":"2024-04-24T01:58:54.154277+0000","last_clean":"2024-04-24T01:58:54.154277+0000","last_became_active":"2024-04-24T01:55:18.219131+0000","last_became_peered":"2024-04-24T01:55:18.219131+0000","last_unstale":"2024-04-24T01:58:54.154277+0000","last_undegraded":"2024-04-24T01:58:54.154277+0000","last_fullsized":"2024-04-24T01:58:54.154277+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"217'58","reported_seq":679,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-04-24T01:58:54.153677+0000","last_change":"2024-04-24T01:58:53.262316+0000","last_active":"2024-04-24T01:58:54.153677+0000","last_peered":"2024-04-24T01:58:54.153677+0000","last_clean":"2024-04-24T01:58:54.153677+0000","last_became_active":"2024-04-24T01:55:30.319958+0000","last_became_peered":"2024-04-24T01:55:30.319958+0000","last_unstale":"2024-04-24T01:58:54.153677+0000","last_undegraded":"2024-04-24T01:58:54.153677+0000","last_fullsized":"2024-04-24T01:58:54.153677+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":5,"snaptrim_duration":0,"stat_sum":{"num_bytes":3428694,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":27,"num_read_kb":19609,"num_write":42,"num_write_kb":11889,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":542,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-04-24T01:58:53.168425+0000","last_change":"2024-04-24T01:55:18.235763+0000","last_active":"2024-04-24T01:58:53.168425+0000","last_peered":"2024-04-24T01:58:53.168425+0000","last_clean":"2024-04-24T01:58:53.168425+0000","last_became_active":"2024-04-24T01:55:18.229175+0000","last_became_peered":"2024-04-24T01:55:18.229175+0000","last_unstale":"2024-04-24T01:58:53.168425+0000","last_undegraded":"2024-04-24T01:58:53.168425+0000","last_fullsized":"2024-04-24T01:58:53.168425+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"212'98","reported_seq":716,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-04-24T01:58:53.160071+0000","last_change":"2024-04-24T01:58:53.178269+0000","last_active":"2024-04-24T01:58:53.160071+0000","last_peered":"2024-04-24T01:58:53.160071+0000","last_clean":"2024-04-24T01:58:53.160071+0000","last_became_active":"2024-04-24T01:55:30.372690+0000","last_became_peered":"2024-04-24T01:55:30.372690+0000","last_unstale":"2024-04-24T01:58:53.160071+0000","last_undegraded":"2024-04-24T01:58:53.160071+0000","last_fullsized":"2024-04-24T01:58:53.160071+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":9,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":46,"num_read_kb":30459,"num_write":40,"num_write_kb":11003,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":531,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-04-24T01:58:54.156633+0000","last_change":"2024-04-24T01:55:18.278351+0000","last_active":"2024-04-24T01:58:54.156633+0000","last_peered":"2024-04-24T01:58:54.156633+0000","last_clean":"2024-04-24T01:58:54.156633+0000","last_became_active":"2024-04-24T01:55:18.273668+0000","last_became_peered":"2024-04-24T01:55:18.273668+0000","last_unstale":"2024-04-24T01:58:54.156633+0000","last_undegraded":"2024-04-24T01:58:54.156633+0000","last_fullsized":"2024-04-24T01:58:54.156633+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"217'227","reported_seq":857,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-04-24T01:58:54.155756+0000","last_change":"2024-04-24T01:58:54.160477+0000","last_active":"2024-04-24T01:58:54.155756+0000","last_peered":"2024-04-24T01:58:54.155756+0000","last_clean":"2024-04-24T01:58:54.155756+0000","last_became_active":"2024-04-24T01:55:30.363123+0000","last_became_peered":"2024-04-24T01:55:30.363123+0000","last_unstale":"2024-04-24T01:58:54.155756+0000","last_undegraded":"2024-04-24T01:58:54.155756+0000","last_fullsized":"2024-04-24T01:58:54.155756+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":227,"log_dups_size":0,"ondisk_log_size":227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":23,"snaptrim_duration":0,"stat_sum":{"num_bytes":11109658,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":157,"num_read_kb":124304,"num_write":126,"num_write_kb":33878,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":535,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-04-24T01:58:52.174004+0000","last_change":"2024-04-24T01:55:18.241117+0000","last_active":"2024-04-24T01:58:52.174004+0000","last_peered":"2024-04-24T01:58:52.174004+0000","last_clean":"2024-04-24T01:58:52.174004+0000","last_became_active":"2024-04-24T01:55:18.234313+0000","last_became_peered":"2024-04-24T01:55:18.234313+0000","last_unstale":"2024-04-24T01:58:52.174004+0000","last_undegraded":"2024-04-24T01:58:52.174004+0000","last_fullsized":"2024-04-24T01:58:52.174004+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":87112430,"num_objects":32,"num_object_clones":3,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1633,"num_read_kb":1128813,"num_write":1476,"num_write_kb":401431,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":156569600,"data_stored":156801095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3099,"ondisk_log_size":3099,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":1}],"osd_stats":[{"osd":3,"up_from":10,"seq":42949673179,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":82832,"kb_used_data":43920,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94289008,"statfs":{"total":96636764160,"available":96551944192,"internally_reserved":0,"allocated":44974080,"data_stored":44056638,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673179,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":70744,"kb_used_data":39392,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94301096,"statfs":{"total":96636764160,"available":96564322304,"internally_reserved":0,"allocated":40337408,"data_stored":40487775,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705884,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":62640,"kb_used_data":28256,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94309200,"statfs":{"total":96636764160,"available":96572620800,"internally_reserved":0,"allocated":28934144,"data_stored":28165257,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705883,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":79216,"kb_used_data":42788,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94292624,"statfs":{"total":96636764160,"available":96555646976,"internally_reserved":0,"allocated":43814912,"data_stored":44154566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":27869184,"data_stored":27583072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":42041344,"data_stored":43384962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40472576,"data_stored":40088323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":46186496,"data_stored":45744738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T01:58:57.731 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T01:58:57.836 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:58:57.834+0000 7f104d99d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:57.840 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:58:57.839+0000 7f104d99d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:58:58.050 INFO:tasks.rados.rados.0.smithi079.stdout:908: done (2 left) 2024-04-24T01:58:58.050 INFO:tasks.rados.rados.0.smithi079.stdout:909: read oid 7 snap -1 2024-04-24T01:58:58.050 INFO:tasks.rados.rados.0.smithi079.stdout:909: expect (ObjNum 375 snap 90 seq_num 375) 2024-04-24T01:58:58.050 INFO:tasks.rados.rados.0.smithi079.stdout:910: snap_remove snap 99 2024-04-24T01:58:59.051 INFO:tasks.rados.rados.0.smithi079.stdout:910: done (3 left) 2024-04-24T01:58:59.051 INFO:tasks.rados.rados.0.smithi079.stdout:911: write oid 28 current snap is 102 2024-04-24T01:58:59.052 INFO:tasks.rados.rados.0.smithi079.stdout:906: finishing write tid 1 to smithi07929762-46 2024-04-24T01:58:59.052 INFO:tasks.rados.rados.0.smithi079.stdout:906: finishing write tid 2 to smithi07929762-46 2024-04-24T01:58:59.052 INFO:tasks.rados.rados.0.smithi079.stdout:906: finishing write tid 3 to smithi07929762-46 2024-04-24T01:58:59.052 INFO:tasks.rados.rados.0.smithi079.stdout:905: finishing write tid 1 to smithi07929762-8 2024-04-24T01:58:59.052 INFO:tasks.rados.rados.0.smithi079.stdout:906: finishing write tid 4 to smithi07929762-46 2024-04-24T01:58:59.052 INFO:tasks.rados.rados.0.smithi079.stdout:906: finishing write tid 5 to smithi07929762-46 2024-04-24T01:58:59.052 INFO:tasks.rados.rados.0.smithi079.stdout:906: oid 46 updating version 0 to 237 2024-04-24T01:58:59.052 INFO:tasks.rados.rados.0.smithi079.stdout:906: oid 46 version 237 is already newer than 234 2024-04-24T01:58:59.052 INFO:tasks.rados.rados.0.smithi079.stdout:906: oid 46 version 237 is already newer than 237 2024-04-24T01:58:59.052 INFO:tasks.rados.rados.0.smithi079.stdout:906: oid 46 version 237 is already newer than 237 2024-04-24T01:58:59.052 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 46 v 237 (ObjNum 448 snap 102 seq_num 448) dirty exists 2024-04-24T01:58:59.052 INFO:tasks.rados.rados.0.smithi079.stdout:906: left oid 46 (ObjNum 448 snap 102 seq_num 448) 2024-04-24T01:58:59.052 INFO:tasks.rados.rados.0.smithi079.stdout:905: finishing write tid 2 to smithi07929762-8 2024-04-24T01:58:59.052 INFO:tasks.rados.rados.0.smithi079.stdout:905: finishing write tid 3 to smithi07929762-8 2024-04-24T01:58:59.052 INFO:tasks.rados.rados.0.smithi079.stdout:911: seq_num 449 ranges {746343=651628,1924415=754807,3397159=149184} 2024-04-24T01:58:59.058 INFO:tasks.rados.rados.0.smithi079.stdout:911: writing smithi07929762-28 from 746343 to 1397971 tid 1 2024-04-24T01:58:59.066 INFO:tasks.rados.rados.0.smithi079.stdout:911: writing smithi07929762-28 from 1924415 to 2679222 tid 2 2024-04-24T01:58:59.071 INFO:tasks.rados.rados.0.smithi079.stdout:911: writing smithi07929762-28 from 3397159 to 3546343 tid 3 2024-04-24T01:58:59.071 INFO:tasks.rados.rados.0.smithi079.stdout:906: done (3 left) 2024-04-24T01:58:59.071 INFO:tasks.rados.rados.0.smithi079.stdout:912: read oid 37 snap -1 2024-04-24T01:58:59.071 INFO:tasks.rados.rados.0.smithi079.stdout:912: expect deleted 2024-04-24T01:58:59.072 INFO:tasks.rados.rados.0.smithi079.stdout:913: delete oid 26 current snap is 102 2024-04-24T01:58:59.072 INFO:tasks.rados.rados.0.smithi079.stdout:905: finishing write tid 4 to smithi07929762-8 2024-04-24T01:58:59.072 INFO:tasks.rados.rados.0.smithi079.stdout:905: finishing write tid 5 to smithi07929762-8 2024-04-24T01:58:59.072 INFO:tasks.rados.rados.0.smithi079.stdout:905: oid 8 updating version 0 to 12 2024-04-24T01:58:59.072 INFO:tasks.rados.rados.0.smithi079.stdout:905: oid 8 version 12 is already newer than 12 2024-04-24T01:58:59.072 INFO:tasks.rados.rados.0.smithi079.stdout:905: oid 8 version 12 is already newer than 12 2024-04-24T01:58:59.072 INFO:tasks.rados.rados.0.smithi079.stdout:905: oid 8 version 12 is already newer than 12 2024-04-24T01:58:59.072 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 8 v 12 (ObjNum 447 snap 102 seq_num 447) dirty exists 2024-04-24T01:58:59.072 INFO:tasks.rados.rados.0.smithi079.stdout:905: left oid 8 (ObjNum 447 snap 102 seq_num 447) 2024-04-24T01:58:59.072 INFO:tasks.rados.rados.0.smithi079.stdout:909: expect (ObjNum 375 snap 90 seq_num 375) 2024-04-24T01:58:59.199 INFO:tasks.rados.rados.0.smithi079.stdout:905: done (4 left) 2024-04-24T01:58:59.199 INFO:tasks.rados.rados.0.smithi079.stdout:909: done (3 left) 2024-04-24T01:58:59.199 INFO:tasks.rados.rados.0.smithi079.stdout:913: done (2 left) 2024-04-24T01:58:59.199 INFO:tasks.rados.rados.0.smithi079.stdout:914: read oid 13 snap -1 2024-04-24T01:58:59.199 INFO:tasks.rados.rados.0.smithi079.stdout:914: expect deleted 2024-04-24T01:58:59.199 INFO:tasks.rados.rados.0.smithi079.stdout:915: write oid 27 current snap is 102 2024-04-24T01:58:59.200 INFO:tasks.rados.rados.0.smithi079.stdout:915: seq_num 451 ranges {779957=402126,1925026=758476,3335652=244305} 2024-04-24T01:58:59.206 INFO:tasks.rados.rados.0.smithi079.stdout:915: writing smithi07929762-27 from 779957 to 1182083 tid 1 2024-04-24T01:58:59.214 INFO:tasks.rados.rados.0.smithi079.stdout:915: writing smithi07929762-27 from 1925026 to 2683502 tid 2 2024-04-24T01:58:59.220 INFO:tasks.rados.rados.0.smithi079.stdout:915: writing smithi07929762-27 from 3335652 to 3579957 tid 3 2024-04-24T01:58:59.220 INFO:tasks.rados.rados.0.smithi079.stdout:916: read oid 25 snap -1 2024-04-24T01:58:59.220 INFO:tasks.rados.rados.0.smithi079.stdout:916: expect (ObjNum 408 snap 97 seq_num 408) 2024-04-24T01:58:59.220 INFO:tasks.rados.rados.0.smithi079.stdout:917: delete oid 14 current snap is 102 2024-04-24T01:58:59.220 INFO:tasks.rados.rados.0.smithi079.stdout:911: finishing write tid 1 to smithi07929762-28 2024-04-24T01:58:59.220 INFO:tasks.rados.rados.0.smithi079.stdout:911: finishing write tid 2 to smithi07929762-28 2024-04-24T01:58:59.220 INFO:tasks.rados.rados.0.smithi079.stdout:911: finishing write tid 3 to smithi07929762-28 2024-04-24T01:58:59.220 INFO:tasks.rados.rados.0.smithi079.stdout:911: finishing write tid 4 to smithi07929762-28 2024-04-24T01:58:59.220 INFO:tasks.rados.rados.0.smithi079.stdout:911: finishing write tid 5 to smithi07929762-28 2024-04-24T01:58:59.220 INFO:tasks.rados.rados.0.smithi079.stdout:911: oid 28 updating version 0 to 272 2024-04-24T01:58:59.220 INFO:tasks.rados.rados.0.smithi079.stdout:911: oid 28 version 272 is already newer than 272 2024-04-24T01:58:59.221 INFO:tasks.rados.rados.0.smithi079.stdout:911: oid 28 version 272 is already newer than 272 2024-04-24T01:58:59.221 INFO:tasks.rados.rados.0.smithi079.stdout:911: oid 28 version 272 is already newer than 272 2024-04-24T01:58:59.221 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 28 v 272 (ObjNum 449 snap 102 seq_num 449) dirty exists 2024-04-24T01:58:59.221 INFO:tasks.rados.rados.0.smithi079.stdout:911: left oid 28 (ObjNum 449 snap 102 seq_num 449) 2024-04-24T01:58:59.232 INFO:tasks.rados.rados.0.smithi079.stdout:911: done (5 left) 2024-04-24T01:58:59.233 INFO:tasks.rados.rados.0.smithi079.stdout:912: done (4 left) 2024-04-24T01:58:59.233 INFO:tasks.rados.rados.0.smithi079.stdout:914: done (3 left) 2024-04-24T01:58:59.233 INFO:tasks.rados.rados.0.smithi079.stdout:917: done (2 left) 2024-04-24T01:58:59.233 INFO:tasks.rados.rados.0.smithi079.stdout:918: read oid 3 snap -1 2024-04-24T01:58:59.233 INFO:tasks.rados.rados.0.smithi079.stdout:918: expect (ObjNum 422 snap 97 seq_num 422) 2024-04-24T01:58:59.233 INFO:tasks.rados.rados.0.smithi079.stdout:919: delete oid 14 current snap is 102 2024-04-24T01:58:59.247 INFO:tasks.rados.rados.0.smithi079.stdout:916: expect (ObjNum 408 snap 97 seq_num 408) 2024-04-24T01:58:59.400 INFO:tasks.rados.rados.0.smithi079.stdout:915: finishing write tid 1 to smithi07929762-27 2024-04-24T01:58:59.400 INFO:tasks.rados.rados.0.smithi079.stdout:918: expect (ObjNum 422 snap 97 seq_num 422) 2024-04-24T01:58:59.516 INFO:tasks.rados.rados.0.smithi079.stdout:915: finishing write tid 2 to smithi07929762-27 2024-04-24T01:58:59.516 INFO:tasks.rados.rados.0.smithi079.stdout:915: finishing write tid 3 to smithi07929762-27 2024-04-24T01:58:59.516 INFO:tasks.rados.rados.0.smithi079.stdout:915: finishing write tid 4 to smithi07929762-27 2024-04-24T01:58:59.516 INFO:tasks.rados.rados.0.smithi079.stdout:915: finishing write tid 5 to smithi07929762-27 2024-04-24T01:58:59.516 INFO:tasks.rados.rados.0.smithi079.stdout:915: oid 27 updating version 0 to 65 2024-04-24T01:58:59.516 INFO:tasks.rados.rados.0.smithi079.stdout:915: oid 27 updating version 65 to 66 2024-04-24T01:58:59.516 INFO:tasks.rados.rados.0.smithi079.stdout:915: oid 27 version 66 is already newer than 66 2024-04-24T01:58:59.516 INFO:tasks.rados.rados.0.smithi079.stdout:915: oid 27 version 66 is already newer than 66 2024-04-24T01:58:59.516 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 27 v 66 (ObjNum 451 snap 102 seq_num 451) dirty exists 2024-04-24T01:58:59.516 INFO:tasks.rados.rados.0.smithi079.stdout:915: left oid 27 (ObjNum 451 snap 102 seq_num 451) 2024-04-24T01:58:59.516 INFO:tasks.rados.rados.0.smithi079.stdout:915: done (3 left) 2024-04-24T01:58:59.516 INFO:tasks.rados.rados.0.smithi079.stdout:916: done (2 left) 2024-04-24T01:58:59.516 INFO:tasks.rados.rados.0.smithi079.stdout:918: done (1 left) 2024-04-24T01:58:59.517 INFO:tasks.rados.rados.0.smithi079.stdout:919: done (0 left) 2024-04-24T01:58:59.517 INFO:tasks.rados.rados.0.smithi079.stdout:920: write (excl) oid 30 current snap is 102 2024-04-24T01:58:59.517 INFO:tasks.rados.rados.0.smithi079.stdout:920: seq_num 454 ranges {430378=627873,1700472=622156,3017921=612457} 2024-04-24T01:58:59.521 INFO:tasks.rados.rados.0.smithi079.stdout:920: writing smithi07929762-30 from 430378 to 1058251 tid 1 2024-04-24T01:58:59.528 INFO:tasks.rados.rados.0.smithi079.stdout:920: writing smithi07929762-30 from 1700472 to 2322628 tid 2 2024-04-24T01:58:59.538 INFO:tasks.rados.rados.0.smithi079.stdout:920: writing smithi07929762-30 from 3017921 to 3630378 tid 3 2024-04-24T01:58:59.538 INFO:tasks.rados.rados.0.smithi079.stdout:921: write (excl) oid 8 current snap is 102 2024-04-24T01:58:59.538 INFO:tasks.rados.rados.0.smithi079.stdout:921: seq_num 455 ranges {447185=703122,1642601=765814,2928991=447707} 2024-04-24T01:58:59.545 INFO:tasks.rados.rados.0.smithi079.stdout:921: writing smithi07929762-8 from 447185 to 1150307 tid 1 2024-04-24T01:58:59.554 INFO:tasks.rados.rados.0.smithi079.stdout:921: writing smithi07929762-8 from 1642601 to 2408415 tid 2 2024-04-24T01:58:59.559 INFO:tasks.rados.rados.0.smithi079.stdout:921: writing smithi07929762-8 from 2928991 to 3376698 tid 3 2024-04-24T01:58:59.560 INFO:tasks.rados.rados.0.smithi079.stdout:922: snap_create 2024-04-24T01:58:59.561 INFO:tasks.rados.rados.0.smithi079.stdout:920: finishing write tid 1 to smithi07929762-30 2024-04-24T01:58:59.573 INFO:tasks.rados.rados.0.smithi079.stdout:920: finishing write tid 2 to smithi07929762-30 2024-04-24T01:58:59.584 INFO:tasks.rados.rados.0.smithi079.stdout:920: finishing write tid 3 to smithi07929762-30 2024-04-24T01:58:59.586 INFO:tasks.rados.rados.0.smithi079.stdout:921: finishing write tid 1 to smithi07929762-8 2024-04-24T01:58:59.586 INFO:tasks.rados.rados.0.smithi079.stdout:920: finishing write tid 4 to smithi07929762-30 2024-04-24T01:58:59.588 INFO:tasks.rados.rados.0.smithi079.stdout:920: finishing write tid 5 to smithi07929762-30 2024-04-24T01:58:59.588 INFO:tasks.rados.rados.0.smithi079.stdout:920: oid 30 updating version 0 to 102 2024-04-24T01:58:59.588 INFO:tasks.rados.rados.0.smithi079.stdout:920: oid 30 version 102 is already newer than 99 2024-04-24T01:58:59.588 INFO:tasks.rados.rados.0.smithi079.stdout:920: oid 30 version 102 is already newer than 102 2024-04-24T01:58:59.588 INFO:tasks.rados.rados.0.smithi079.stdout:920: oid 30 version 102 is already newer than 102 2024-04-24T01:58:59.588 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 30 v 102 (ObjNum 454 snap 102 seq_num 454) dirty exists 2024-04-24T01:58:59.588 INFO:tasks.rados.rados.0.smithi079.stdout:920: left oid 30 (ObjNum 454 snap 102 seq_num 454) 2024-04-24T01:58:59.607 INFO:tasks.rados.rados.0.smithi079.stdout:921: finishing write tid 2 to smithi07929762-8 2024-04-24T01:58:59.614 INFO:tasks.rados.rados.0.smithi079.stdout:921: finishing write tid 3 to smithi07929762-8 2024-04-24T01:58:59.618 INFO:tasks.rados.rados.0.smithi079.stdout:921: finishing write tid 4 to smithi07929762-8 2024-04-24T01:58:59.619 INFO:tasks.rados.rados.0.smithi079.stdout:921: finishing write tid 5 to smithi07929762-8 2024-04-24T01:58:59.619 INFO:tasks.rados.rados.0.smithi079.stdout:921: oid 8 updating version 0 to 16 2024-04-24T01:58:59.619 INFO:tasks.rados.rados.0.smithi079.stdout:921: oid 8 version 16 is already newer than 16 2024-04-24T01:58:59.619 INFO:tasks.rados.rados.0.smithi079.stdout:921: oid 8 version 16 is already newer than 13 2024-04-24T01:58:59.619 INFO:tasks.rados.rados.0.smithi079.stdout:921: oid 8 version 16 is already newer than 16 2024-04-24T01:58:59.619 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 8 v 16 (ObjNum 455 snap 102 seq_num 455) dirty exists 2024-04-24T01:58:59.619 INFO:tasks.rados.rados.0.smithi079.stdout:921: left oid 8 (ObjNum 455 snap 102 seq_num 455) 2024-04-24T01:58:59.936 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T01:59:00.041 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:59:00.038+0000 7fad5c66d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:59:00.044 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:59:00.044+0000 7fad5c66d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:59:00.062 INFO:tasks.rados.rados.0.smithi079.stdout:920: done (2 left) 2024-04-24T01:59:00.062 INFO:tasks.rados.rados.0.smithi079.stdout:921: done (1 left) 2024-04-24T01:59:00.062 INFO:tasks.rados.rados.0.smithi079.stdout:922: done (0 left) 2024-04-24T01:59:00.062 INFO:tasks.rados.rados.0.smithi079.stdout:923: read oid 48 snap -1 2024-04-24T01:59:00.062 INFO:tasks.rados.rados.0.smithi079.stdout:923: expect deleted 2024-04-24T01:59:00.062 INFO:tasks.rados.rados.0.smithi079.stdout:924: delete oid 38 current snap is 103 2024-04-24T01:59:00.120 INFO:tasks.rados.rados.0.smithi079.stdout:923: done (1 left) 2024-04-24T01:59:00.120 INFO:tasks.rados.rados.0.smithi079.stdout:924: done (0 left) 2024-04-24T01:59:00.120 INFO:tasks.rados.rados.0.smithi079.stdout:925: write oid 33 current snap is 103 2024-04-24T01:59:00.120 INFO:tasks.rados.rados.0.smithi079.stdout:925: seq_num 457 ranges {480799=569973,1475918=769483,2700190=586604} 2024-04-24T01:59:00.125 INFO:tasks.rados.rados.0.smithi079.stdout:925: writing smithi07929762-33 from 480799 to 1050772 tid 1 2024-04-24T01:59:00.132 INFO:tasks.rados.rados.0.smithi079.stdout:925: writing smithi07929762-33 from 1475918 to 2245401 tid 2 2024-04-24T01:59:00.139 INFO:tasks.rados.rados.0.smithi079.stdout:925: writing smithi07929762-33 from 2700190 to 3286794 tid 3 2024-04-24T01:59:00.139 INFO:tasks.rados.rados.0.smithi079.stdout:926: write oid 26 current snap is 103 2024-04-24T01:59:00.139 INFO:tasks.rados.rados.0.smithi079.stdout:926: seq_num 458 ranges {497606=645222,1934400=629494,3243966=453640} 2024-04-24T01:59:00.145 INFO:tasks.rados.rados.0.smithi079.stdout:926: writing smithi07929762-26 from 497606 to 1142828 tid 1 2024-04-24T01:59:00.153 INFO:tasks.rados.rados.0.smithi079.stdout:926: writing smithi07929762-26 from 1934400 to 2563894 tid 2 2024-04-24T01:59:00.159 INFO:tasks.rados.rados.0.smithi079.stdout:926: writing smithi07929762-26 from 3243966 to 3697606 tid 3 2024-04-24T01:59:00.160 INFO:tasks.rados.rados.0.smithi079.stdout:927: write (excl) oid 27 current snap is 103 2024-04-24T01:59:00.160 INFO:tasks.rados.rados.0.smithi079.stdout:927: seq_num 459 ranges {514413=720471,1876529=773152,3438683=275730} 2024-04-24T01:59:00.166 INFO:tasks.rados.rados.0.smithi079.stdout:927: writing smithi07929762-27 from 514413 to 1234884 tid 1 2024-04-24T01:59:00.174 INFO:tasks.rados.rados.0.smithi079.stdout:927: writing smithi07929762-27 from 1876529 to 2649681 tid 2 2024-04-24T01:59:00.181 INFO:tasks.rados.rados.0.smithi079.stdout:927: writing smithi07929762-27 from 3438683 to 3714413 tid 3 2024-04-24T01:59:00.181 INFO:tasks.rados.rados.0.smithi079.stdout:928: write (excl) oid 19 current snap is 103 2024-04-24T01:59:00.181 INFO:tasks.rados.rados.0.smithi079.stdout:928: seq_num 460 ranges {531220=795720,1935011=516810,3066106=420420} 2024-04-24T01:59:00.189 INFO:tasks.rados.rados.0.smithi079.stdout:928: writing smithi07929762-19 from 531220 to 1326940 tid 1 2024-04-24T01:59:00.202 INFO:tasks.rados.rados.0.smithi079.stdout:928: writing smithi07929762-19 from 1935011 to 2451821 tid 2 2024-04-24T01:59:00.210 INFO:tasks.rados.rados.0.smithi079.stdout:928: writing smithi07929762-19 from 3066106 to 3486526 tid 3 2024-04-24T01:59:00.210 INFO:tasks.rados.rados.0.smithi079.stdout:929: delete oid 21 current snap is 103 2024-04-24T01:59:00.238 INFO:tasks.rados.rados.0.smithi079.stdout:925: finishing write tid 1 to smithi07929762-33 2024-04-24T01:59:00.238 INFO:tasks.rados.rados.0.smithi079.stdout:926: finishing write tid 1 to smithi07929762-26 2024-04-24T01:59:00.271 INFO:tasks.rados.rados.0.smithi079.stdout:925: finishing write tid 2 to smithi07929762-33 2024-04-24T01:59:00.282 INFO:tasks.rados.rados.0.smithi079.stdout:926: finishing write tid 2 to smithi07929762-26 2024-04-24T01:59:00.326 INFO:tasks.rados.rados.0.smithi079.stdout:927: finishing write tid 1 to smithi07929762-27 2024-04-24T01:59:00.348 INFO:tasks.rados.rados.0.smithi079.stdout:925: finishing write tid 3 to smithi07929762-33 2024-04-24T01:59:00.354 INFO:tasks.rados.rados.0.smithi079.stdout:929: done (4 left) 2024-04-24T01:59:00.354 INFO:tasks.rados.rados.0.smithi079.stdout:930: delete oid 12 current snap is 103 2024-04-24T01:59:00.354 INFO:tasks.rados.rados.0.smithi079.stdout:928: finishing write tid 1 to smithi07929762-19 2024-04-24T01:59:00.360 INFO:tasks.rados.rados.0.smithi079.stdout:926: finishing write tid 3 to smithi07929762-26 2024-04-24T01:59:00.369 INFO:tasks.rados.rados.0.smithi079.stdout:926: finishing write tid 4 to smithi07929762-26 2024-04-24T01:59:00.371 INFO:tasks.rados.rados.0.smithi079.stdout:926: finishing write tid 5 to smithi07929762-26 2024-04-24T01:59:00.372 INFO:tasks.rados.rados.0.smithi079.stdout:926: oid 26 updating version 0 to 344 2024-04-24T01:59:00.372 INFO:tasks.rados.rados.0.smithi079.stdout:926: oid 26 version 344 is already newer than 344 2024-04-24T01:59:00.372 INFO:tasks.rados.rados.0.smithi079.stdout:926: oid 26 version 344 is already newer than 344 2024-04-24T01:59:00.372 INFO:tasks.rados.rados.0.smithi079.stdout:926: oid 26 version 344 is already newer than 344 2024-04-24T01:59:00.372 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 26 v 344 (ObjNum 458 snap 103 seq_num 458) dirty exists 2024-04-24T01:59:00.372 INFO:tasks.rados.rados.0.smithi079.stdout:926: left oid 26 (ObjNum 458 snap 103 seq_num 458) 2024-04-24T01:59:00.387 INFO:tasks.rados.rados.0.smithi079.stdout:927: finishing write tid 2 to smithi07929762-27 2024-04-24T01:59:00.393 INFO:tasks.rados.rados.0.smithi079.stdout:925: finishing write tid 4 to smithi07929762-33 2024-04-24T01:59:00.394 INFO:tasks.rados.rados.0.smithi079.stdout:927: finishing write tid 3 to smithi07929762-27 2024-04-24T01:59:00.396 INFO:tasks.rados.rados.0.smithi079.stdout:925: finishing write tid 5 to smithi07929762-33 2024-04-24T01:59:00.396 INFO:tasks.rados.rados.0.smithi079.stdout:925: oid 33 updating version 0 to 304 2024-04-24T01:59:00.396 INFO:tasks.rados.rados.0.smithi079.stdout:925: oid 33 version 304 is already newer than 304 2024-04-24T01:59:00.396 INFO:tasks.rados.rados.0.smithi079.stdout:925: oid 33 version 304 is already newer than 304 2024-04-24T01:59:00.396 INFO:tasks.rados.rados.0.smithi079.stdout:925: oid 33 version 304 is already newer than 304 2024-04-24T01:59:00.396 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 33 v 304 (ObjNum 457 snap 103 seq_num 457) dirty exists 2024-04-24T01:59:00.396 INFO:tasks.rados.rados.0.smithi079.stdout:925: left oid 33 (ObjNum 457 snap 103 seq_num 457) 2024-04-24T01:59:00.398 INFO:tasks.rados.rados.0.smithi079.stdout:928: finishing write tid 2 to smithi07929762-19 2024-04-24T01:59:00.403 INFO:tasks.rados.rados.0.smithi079.stdout:927: finishing write tid 4 to smithi07929762-27 2024-04-24T01:59:00.406 INFO:tasks.rados.rados.0.smithi079.stdout:927: finishing write tid 5 to smithi07929762-27 2024-04-24T01:59:00.406 INFO:tasks.rados.rados.0.smithi079.stdout:927: oid 27 updating version 0 to 71 2024-04-24T01:59:00.406 INFO:tasks.rados.rados.0.smithi079.stdout:927: oid 27 version 71 is already newer than 71 2024-04-24T01:59:00.406 INFO:tasks.rados.rados.0.smithi079.stdout:927: oid 27 version 71 is already newer than 68 2024-04-24T01:59:00.407 INFO:tasks.rados.rados.0.smithi079.stdout:927: oid 27 version 71 is already newer than 71 2024-04-24T01:59:00.407 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 27 v 71 (ObjNum 459 snap 103 seq_num 459) dirty exists 2024-04-24T01:59:00.407 INFO:tasks.rados.rados.0.smithi079.stdout:927: left oid 27 (ObjNum 459 snap 103 seq_num 459) 2024-04-24T01:59:00.409 INFO:tasks.rados.rados.0.smithi079.stdout:928: finishing write tid 3 to smithi07929762-19 2024-04-24T01:59:00.412 INFO:tasks.rados.rados.0.smithi079.stdout:928: finishing write tid 4 to smithi07929762-19 2024-04-24T01:59:00.413 INFO:tasks.rados.rados.0.smithi079.stdout:925: done (4 left) 2024-04-24T01:59:00.413 INFO:tasks.rados.rados.0.smithi079.stdout:926: done (3 left) 2024-04-24T01:59:00.413 INFO:tasks.rados.rados.0.smithi079.stdout:927: done (2 left) 2024-04-24T01:59:00.413 INFO:tasks.rados.rados.0.smithi079.stdout:930: done (1 left) 2024-04-24T01:59:00.413 INFO:tasks.rados.rados.0.smithi079.stdout:931: snap_create 2024-04-24T01:59:00.414 INFO:tasks.rados.rados.0.smithi079.stdout:928: finishing write tid 5 to smithi07929762-19 2024-04-24T01:59:00.414 INFO:tasks.rados.rados.0.smithi079.stdout:928: oid 19 updating version 0 to 276 2024-04-24T01:59:00.414 INFO:tasks.rados.rados.0.smithi079.stdout:928: oid 19 updating version 276 to 279 2024-04-24T01:59:00.414 INFO:tasks.rados.rados.0.smithi079.stdout:928: oid 19 version 279 is already newer than 279 2024-04-24T01:59:00.415 INFO:tasks.rados.rados.0.smithi079.stdout:928: oid 19 version 279 is already newer than 279 2024-04-24T01:59:00.415 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 19 v 279 (ObjNum 460 snap 103 seq_num 460) dirty exists 2024-04-24T01:59:00.415 INFO:tasks.rados.rados.0.smithi079.stdout:928: left oid 19 (ObjNum 460 snap 103 seq_num 460) 2024-04-24T01:59:01.070 INFO:tasks.rados.rados.0.smithi079.stdout:928: done (1 left) 2024-04-24T01:59:01.071 INFO:tasks.rados.rados.0.smithi079.stdout:931: done (0 left) 2024-04-24T01:59:01.071 INFO:tasks.rados.rados.0.smithi079.stdout:932: read oid 42 snap -1 2024-04-24T01:59:01.071 INFO:tasks.rados.rados.0.smithi079.stdout:932: expect (ObjNum 434 snap 99 seq_num 434) 2024-04-24T01:59:01.071 INFO:tasks.rados.rados.0.smithi079.stdout:933: read oid 13 snap 102 2024-04-24T01:59:01.071 INFO:tasks.rados.rados.0.smithi079.stdout:933: expect deleted 2024-04-24T01:59:01.071 INFO:tasks.rados.rados.0.smithi079.stdout:934: read oid 41 snap 103 2024-04-24T01:59:01.071 INFO:tasks.rados.rados.0.smithi079.stdout:934: expect (ObjNum 370 snap 90 seq_num 370) 2024-04-24T01:59:01.071 INFO:tasks.rados.rados.0.smithi079.stdout:935: write (excl) oid 48 current snap is 104 2024-04-24T01:59:01.071 INFO:tasks.rados.rados.0.smithi079.stdout:935: seq_num 463 ranges {581641=621467,1994104=780490,3432022=349619} 2024-04-24T01:59:01.079 INFO:tasks.rados.rados.0.smithi079.stdout:935: writing smithi07929762-48 from 581641 to 1203108 tid 1 2024-04-24T01:59:01.089 INFO:tasks.rados.rados.0.smithi079.stdout:935: writing smithi07929762-48 from 1994104 to 2774594 tid 2 2024-04-24T01:59:01.096 INFO:tasks.rados.rados.0.smithi079.stdout:935: writing smithi07929762-48 from 3432022 to 3781641 tid 3 2024-04-24T01:59:01.096 INFO:tasks.rados.rados.0.smithi079.stdout:936: write (excl) oid 8 current snap is 104 2024-04-24T01:59:01.096 INFO:tasks.rados.rados.0.smithi079.stdout:936: seq_num 464 ranges {598448=413069,1768939=524148,2775798=581861,3781488=16960} 2024-04-24T01:59:01.103 INFO:tasks.rados.rados.0.smithi079.stdout:936: writing smithi07929762-8 from 598448 to 1011517 tid 1 2024-04-24T01:59:01.113 INFO:tasks.rados.rados.0.smithi079.stdout:936: writing smithi07929762-8 from 1768939 to 2293087 tid 2 2024-04-24T01:59:01.119 INFO:tasks.rados.rados.0.smithi079.stdout:936: writing smithi07929762-8 from 2775798 to 3357659 tid 3 2024-04-24T01:59:01.122 INFO:tasks.rados.rados.0.smithi079.stdout:936: writing smithi07929762-8 from 3781488 to 3798448 tid 4 2024-04-24T01:59:01.122 INFO:tasks.rados.rados.0.smithi079.stdout:937: snap_create 2024-04-24T01:59:01.126 INFO:tasks.rados.rados.0.smithi079.stdout:934: expect (ObjNum 370 snap 90 seq_num 370) 2024-04-24T01:59:01.225 INFO:tasks.rados.rados.0.smithi079.stdout:935: finishing write tid 1 to smithi07929762-48 2024-04-24T01:59:01.225 INFO:tasks.rados.rados.0.smithi079.stdout:935: finishing write tid 2 to smithi07929762-48 2024-04-24T01:59:01.225 INFO:tasks.rados.rados.0.smithi079.stdout:932: expect (ObjNum 434 snap 99 seq_num 434) 2024-04-24T01:59:01.428 INFO:tasks.rados.rados.0.smithi079.stdout:935: finishing write tid 3 to smithi07929762-48 2024-04-24T01:59:01.428 INFO:tasks.rados.rados.0.smithi079.stdout:936: finishing write tid 1 to smithi07929762-8 2024-04-24T01:59:01.428 INFO:tasks.rados.rados.0.smithi079.stdout:936: finishing write tid 2 to smithi07929762-8 2024-04-24T01:59:01.429 INFO:tasks.rados.rados.0.smithi079.stdout:935: finishing write tid 4 to smithi07929762-48 2024-04-24T01:59:01.429 INFO:tasks.rados.rados.0.smithi079.stdout:935: finishing write tid 5 to smithi07929762-48 2024-04-24T01:59:01.429 INFO:tasks.rados.rados.0.smithi079.stdout:935: oid 48 updating version 0 to 166 2024-04-24T01:59:01.429 INFO:tasks.rados.rados.0.smithi079.stdout:935: oid 48 version 166 is already newer than 166 2024-04-24T01:59:01.429 INFO:tasks.rados.rados.0.smithi079.stdout:935: oid 48 version 166 is already newer than 166 2024-04-24T01:59:01.429 INFO:tasks.rados.rados.0.smithi079.stdout:935: oid 48 version 166 is already newer than 163 2024-04-24T01:59:01.429 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 48 v 166 (ObjNum 463 snap 104 seq_num 463) dirty exists 2024-04-24T01:59:01.429 INFO:tasks.rados.rados.0.smithi079.stdout:935: left oid 48 (ObjNum 463 snap 104 seq_num 463) 2024-04-24T01:59:01.429 INFO:tasks.rados.rados.0.smithi079.stdout:936: finishing write tid 3 to smithi07929762-8 2024-04-24T01:59:01.429 INFO:tasks.rados.rados.0.smithi079.stdout:936: finishing write tid 4 to smithi07929762-8 2024-04-24T01:59:01.429 INFO:tasks.rados.rados.0.smithi079.stdout:936: finishing write tid 5 to smithi07929762-8 2024-04-24T01:59:01.429 INFO:tasks.rados.rados.0.smithi079.stdout:936: finishing write tid 6 to smithi07929762-8 2024-04-24T01:59:01.429 INFO:tasks.rados.rados.0.smithi079.stdout:936: oid 8 updating version 0 to 22 2024-04-24T01:59:01.430 INFO:tasks.rados.rados.0.smithi079.stdout:936: oid 8 version 22 is already newer than 22 2024-04-24T01:59:01.430 INFO:tasks.rados.rados.0.smithi079.stdout:936: oid 8 version 22 is already newer than 22 2024-04-24T01:59:01.430 INFO:tasks.rados.rados.0.smithi079.stdout:936: oid 8 version 22 is already newer than 22 2024-04-24T01:59:01.430 INFO:tasks.rados.rados.0.smithi079.stdout:936: oid 8 version 22 is already newer than 18 2024-04-24T01:59:01.430 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 8 v 22 (ObjNum 464 snap 104 seq_num 464) dirty exists 2024-04-24T01:59:01.430 INFO:tasks.rados.rados.0.smithi079.stdout:936: left oid 8 (ObjNum 464 snap 104 seq_num 464) 2024-04-24T01:59:02.072 INFO:tasks.rados.rados.0.smithi079.stdout:932: done (5 left) 2024-04-24T01:59:02.073 INFO:tasks.rados.rados.0.smithi079.stdout:933: done (4 left) 2024-04-24T01:59:02.073 INFO:tasks.rados.rados.0.smithi079.stdout:934: done (3 left) 2024-04-24T01:59:02.073 INFO:tasks.rados.rados.0.smithi079.stdout:935: done (2 left) 2024-04-24T01:59:02.073 INFO:tasks.rados.rados.0.smithi079.stdout:936: done (1 left) 2024-04-24T01:59:02.073 INFO:tasks.rados.rados.0.smithi079.stdout:937: done (0 left) 2024-04-24T01:59:02.073 INFO:tasks.rados.rados.0.smithi079.stdout:938: snap_create 2024-04-24T01:59:02.174 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T01:59:02.299 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:59:02.296+0000 7fdefaaed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:59:02.305 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:59:02.304+0000 7fdefaaed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:59:02.474 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-04-24T01:59:02.474 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-04-24T01:59:02.474 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-04-24T01:59:02.475 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-04-24T01:59:02.584 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:59:02.578+0000 7fdcefc2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:59:02.587 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:59:02.587+0000 7fdcefc2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:59:02.601 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T01:59:02.601 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T01:59:02.602 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T01:59:02.602 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T01:59:02.602 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T01:59:02.602 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T01:59:02.602 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T01:59:02.602 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T01:59:02.602 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T01:59:02.602 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T01:59:02.602 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T01:59:02.602 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T01:59:02.604 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T01:59:02.605 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T01:59:02.605 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-04-24T01:59:02.712 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:59:02.706+0000 7f824751d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:59:02.716 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:59:02.715+0000 7f824751d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:59:02.730 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T01:59:02.730 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T01:59:02.730 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T01:59:02.730 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T01:59:02.730 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T01:59:02.730 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T01:59:02.730 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T01:59:02.731 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T01:59:02.731 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T01:59:02.731 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T01:59:02.731 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T01:59:02.731 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T01:59:02.733 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T01:59:02.733 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T01:59:02.733 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-04-24T01:59:02.841 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:59:02.835+0000 7f2c3906d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:59:02.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:59:02.843+0000 7f2c3906d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:59:02.860 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T01:59:02.861 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T01:59:02.861 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T01:59:02.861 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T01:59:02.861 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T01:59:02.861 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T01:59:02.861 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T01:59:02.861 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T01:59:02.861 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T01:59:02.861 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T01:59:02.861 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T01:59:02.861 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T01:59:02.863 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T01:59:02.864 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T01:59:02.864 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-04-24T01:59:02.971 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:59:02.965+0000 7ff82c59d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:59:02.975 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:59:02.974+0000 7ff82c59d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:59:02.991 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T01:59:02.991 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T01:59:02.991 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T01:59:02.991 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T01:59:02.991 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T01:59:02.991 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T01:59:02.991 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T01:59:02.992 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T01:59:02.992 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T01:59:02.992 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T01:59:02.992 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T01:59:02.992 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T01:59:02.994 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T01:59:02.994 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T01:59:03.077 INFO:tasks.rados.rados.0.smithi079.stdout:938: done (0 left) 2024-04-24T01:59:03.077 INFO:tasks.rados.rados.0.smithi079.stdout:939: write (excl) oid 44 current snap is 106 2024-04-24T01:59:03.077 INFO:tasks.rados.rados.0.smithi079.stdout:939: seq_num 465 ranges {615255=488318,1827421=784159,3203221=612034} 2024-04-24T01:59:03.082 INFO:tasks.rados.rados.0.smithi079.stdout:939: writing smithi07929762-44 from 615255 to 1103573 tid 1 2024-04-24T01:59:03.091 INFO:tasks.rados.rados.0.smithi079.stdout:939: writing smithi07929762-44 from 1827421 to 2611580 tid 2 2024-04-24T01:59:03.098 INFO:tasks.rados.rados.0.smithi079.stdout:939: writing smithi07929762-44 from 3203221 to 3815255 tid 3 2024-04-24T01:59:03.098 INFO:tasks.rados.rados.0.smithi079.stdout:940: read oid 11 snap 104 2024-04-24T01:59:03.098 INFO:tasks.rados.rados.0.smithi079.stdout:940: expect (ObjNum 396 snap 94 seq_num 396) 2024-04-24T01:59:03.098 INFO:tasks.rados.rados.0.smithi079.stdout:941: read oid 48 snap -1 2024-04-24T01:59:03.098 INFO:tasks.rados.rados.0.smithi079.stdout:941: expect (ObjNum 463 snap 104 seq_num 463) 2024-04-24T01:59:03.098 INFO:tasks.rados.rados.0.smithi079.stdout:942: snap_remove snap 102 2024-04-24T01:59:04.079 INFO:tasks.rados.rados.0.smithi079.stdout:942: done (3 left) 2024-04-24T01:59:04.079 INFO:tasks.rados.rados.0.smithi079.stdout:943: snap_remove snap 105 2024-04-24T01:59:04.398 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T01:59:04.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:59:04.505+0000 7f994d02d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:59:04.511 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:59:04.510+0000 7f994d02d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:59:05.083 INFO:tasks.rados.rados.0.smithi079.stdout:943: done (3 left) 2024-04-24T01:59:05.083 INFO:tasks.rados.rados.0.smithi079.stdout:944: write oid 19 current snap is 106 2024-04-24T01:59:05.083 INFO:tasks.rados.rados.0.smithi079.stdout:944: seq_num 466 ranges {632062=563567,1885903=527817,2830644=720758} 2024-04-24T01:59:05.089 INFO:tasks.rados.rados.0.smithi079.stdout:944: writing smithi07929762-19 from 632062 to 1195629 tid 1 2024-04-24T01:59:05.096 INFO:tasks.rados.rados.0.smithi079.stdout:944: writing smithi07929762-19 from 1885903 to 2413720 tid 2 2024-04-24T01:59:05.102 INFO:tasks.rados.rados.0.smithi079.stdout:944: writing smithi07929762-19 from 2830644 to 3551402 tid 3 2024-04-24T01:59:05.102 INFO:tasks.rados.rados.0.smithi079.stdout:945: snap_remove snap 103 2024-04-24T01:59:06.044 INFO:tasks.rados.rados.0.smithi079.stdout:945: done (4 left) 2024-04-24T01:59:06.045 INFO:tasks.rados.rados.0.smithi079.stdout:946: snap_create 2024-04-24T01:59:06.045 INFO:tasks.rados.rados.0.smithi079.stdout:939: finishing write tid 1 to smithi07929762-44 2024-04-24T01:59:06.045 INFO:tasks.rados.rados.0.smithi079.stdout:940: expect (ObjNum 396 snap 94 seq_num 396) 2024-04-24T01:59:06.146 INFO:tasks.rados.rados.0.smithi079.stdout:941: expect (ObjNum 463 snap 104 seq_num 463) 2024-04-24T01:59:06.319 INFO:tasks.rados.rados.0.smithi079.stdout:939: finishing write tid 2 to smithi07929762-44 2024-04-24T01:59:06.319 INFO:tasks.rados.rados.0.smithi079.stdout:939: finishing write tid 3 to smithi07929762-44 2024-04-24T01:59:06.319 INFO:tasks.rados.rados.0.smithi079.stdout:939: finishing write tid 4 to smithi07929762-44 2024-04-24T01:59:06.319 INFO:tasks.rados.rados.0.smithi079.stdout:939: finishing write tid 5 to smithi07929762-44 2024-04-24T01:59:06.320 INFO:tasks.rados.rados.0.smithi079.stdout:939: oid 44 updating version 0 to 229 2024-04-24T01:59:06.320 INFO:tasks.rados.rados.0.smithi079.stdout:939: oid 44 version 229 is already newer than 229 2024-04-24T01:59:06.320 INFO:tasks.rados.rados.0.smithi079.stdout:939: oid 44 version 229 is already newer than 226 2024-04-24T01:59:06.320 INFO:tasks.rados.rados.0.smithi079.stdout:939: oid 44 version 229 is already newer than 229 2024-04-24T01:59:06.320 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 44 v 229 (ObjNum 465 snap 106 seq_num 465) dirty exists 2024-04-24T01:59:06.320 INFO:tasks.rados.rados.0.smithi079.stdout:939: left oid 44 (ObjNum 465 snap 106 seq_num 465) 2024-04-24T01:59:06.320 INFO:tasks.rados.rados.0.smithi079.stdout:944: finishing write tid 1 to smithi07929762-19 2024-04-24T01:59:06.320 INFO:tasks.rados.rados.0.smithi079.stdout:944: finishing write tid 2 to smithi07929762-19 2024-04-24T01:59:06.320 INFO:tasks.rados.rados.0.smithi079.stdout:944: finishing write tid 3 to smithi07929762-19 2024-04-24T01:59:06.320 INFO:tasks.rados.rados.0.smithi079.stdout:944: finishing write tid 4 to smithi07929762-19 2024-04-24T01:59:06.320 INFO:tasks.rados.rados.0.smithi079.stdout:944: finishing write tid 5 to smithi07929762-19 2024-04-24T01:59:06.320 INFO:tasks.rados.rados.0.smithi079.stdout:944: oid 19 updating version 0 to 286 2024-04-24T01:59:06.320 INFO:tasks.rados.rados.0.smithi079.stdout:944: oid 19 version 286 is already newer than 286 2024-04-24T01:59:06.320 INFO:tasks.rados.rados.0.smithi079.stdout:944: oid 19 version 286 is already newer than 286 2024-04-24T01:59:06.320 INFO:tasks.rados.rados.0.smithi079.stdout:944: oid 19 version 286 is already newer than 286 2024-04-24T01:59:06.320 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 19 v 286 (ObjNum 466 snap 106 seq_num 466) dirty exists 2024-04-24T01:59:06.320 INFO:tasks.rados.rados.0.smithi079.stdout:944: left oid 19 (ObjNum 466 snap 106 seq_num 466) 2024-04-24T01:59:06.605 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T01:59:06.710 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:59:06.708+0000 7f5e7219d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:59:06.714 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:59:06.713+0000 7f5e7219d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:59:07.045 INFO:tasks.rados.rados.0.smithi079.stdout:939: done (4 left) 2024-04-24T01:59:07.045 INFO:tasks.rados.rados.0.smithi079.stdout:940: done (3 left) 2024-04-24T01:59:07.046 INFO:tasks.rados.rados.0.smithi079.stdout:941: done (2 left) 2024-04-24T01:59:07.046 INFO:tasks.rados.rados.0.smithi079.stdout:944: done (1 left) 2024-04-24T01:59:07.046 INFO:tasks.rados.rados.0.smithi079.stdout:946: done (0 left) 2024-04-24T01:59:07.046 INFO:tasks.rados.rados.0.smithi079.stdout:947: snap_remove snap 104 2024-04-24T01:59:07.995 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-04-24T01:59:07.996 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-04-24T01:59:07.996 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T01:59:08.049 INFO:tasks.rados.rados.0.smithi079.stdout:947: done (0 left) 2024-04-24T01:59:08.049 INFO:tasks.rados.rados.0.smithi079.stdout:948: delete oid 19 current snap is 107 2024-04-24T01:59:08.117 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:59:08.112+0000 7f123439d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:59:08.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:59:08.122+0000 7f123439d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:59:08.141 INFO:tasks.rados.rados.0.smithi079.stdout:948: done (0 left) 2024-04-24T01:59:08.141 INFO:tasks.rados.rados.0.smithi079.stdout:949: write oid 48 current snap is 107 2024-04-24T01:59:08.141 INFO:tasks.rados.rados.0.smithi079.stdout:949: seq_num 468 ranges {665676=714065,1886514=531486,3169137=696539} 2024-04-24T01:59:08.148 INFO:tasks.rados.rados.0.smithi079.stdout:949: writing smithi07929762-48 from 665676 to 1379741 tid 1 2024-04-24T01:59:08.154 INFO:tasks.rados.rados.0.smithi079.stdout:949: writing smithi07929762-48 from 1886514 to 2418000 tid 2 2024-04-24T01:59:08.162 INFO:tasks.rados.rados.0.smithi079.stdout:949: writing smithi07929762-48 from 3169137 to 3865676 tid 3 2024-04-24T01:59:08.163 INFO:tasks.rados.rados.0.smithi079.stdout:950: delete oid 50 current snap is 107 2024-04-24T01:59:08.235 INFO:tasks.rados.rados.0.smithi079.stdout:949: finishing write tid 1 to smithi07929762-48 2024-04-24T01:59:08.256 INFO:tasks.rados.rados.0.smithi079.stdout:949: finishing write tid 2 to smithi07929762-48 2024-04-24T01:59:08.261 INFO:tasks.rados.rados.0.smithi079.stdout:949: finishing write tid 3 to smithi07929762-48 2024-04-24T01:59:08.265 INFO:tasks.rados.rados.0.smithi079.stdout:949: finishing write tid 4 to smithi07929762-48 2024-04-24T01:59:08.266 INFO:tasks.rados.rados.0.smithi079.stdout:949: finishing write tid 5 to smithi07929762-48 2024-04-24T01:59:08.266 INFO:tasks.rados.rados.0.smithi079.stdout:949: oid 48 updating version 0 to 171 2024-04-24T01:59:08.266 INFO:tasks.rados.rados.0.smithi079.stdout:949: oid 48 version 171 is already newer than 171 2024-04-24T01:59:08.266 INFO:tasks.rados.rados.0.smithi079.stdout:949: oid 48 version 171 is already newer than 171 2024-04-24T01:59:08.266 INFO:tasks.rados.rados.0.smithi079.stdout:949: oid 48 version 171 is already newer than 171 2024-04-24T01:59:08.266 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 48 v 171 (ObjNum 468 snap 107 seq_num 468) dirty exists 2024-04-24T01:59:08.267 INFO:tasks.rados.rados.0.smithi079.stdout:949: left oid 48 (ObjNum 468 snap 107 seq_num 468) 2024-04-24T01:59:08.269 INFO:tasks.rados.rados.0.smithi079.stdout:949: done (1 left) 2024-04-24T01:59:08.269 INFO:tasks.rados.rados.0.smithi079.stdout:950: done (0 left) 2024-04-24T01:59:08.269 INFO:tasks.rados.rados.0.smithi079.stdout:951: snap_remove snap 106 2024-04-24T01:59:08.336 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:59:08.336 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-24T01:59:08.349 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":340,"stamp":"2024-04-24T01:59:08.046848+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":95097911,"num_objects":39,"num_object_clones":5,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":1811,"num_read_kb":1183017,"num_write":1691,"num_write_kb":426456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3295,"ondisk_log_size":3295,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":338668,"kb_used_data":202252,"kb_used_omap":0,"kb_used_meta":136384,"kb_avail":377148692,"statfs":{"total":386547056640,"available":386200260608,"internally_reserved":0,"allocated":207106048,"data_stored":210025173,"data_compressed":89889,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":139657216},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":7495988,"num_objects":7,"num_object_clones":3,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":33,"num_read_kb":23490,"num_write":33,"num_write_kb":10171,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.968346"},"pg_stats":[{"pgid":"3.f","version":"225'166","reported_seq":838,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-24T01:59:03.162210+0000","last_change":"2024-04-24T01:59:00.089118+0000","last_active":"2024-04-24T01:59:03.162210+0000","last_peered":"2024-04-24T01:59:03.162210+0000","last_clean":"2024-04-24T01:59:03.162210+0000","last_became_active":"2024-04-24T01:55:30.386726+0000","last_became_peered":"2024-04-24T01:55:30.386726+0000","last_unstale":"2024-04-24T01:59:03.162210+0000","last_undegraded":"2024-04-24T01:59:03.162210+0000","last_fullsized":"2024-04-24T01:59:03.162210+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":166,"log_dups_size":0,"ondisk_log_size":166,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":16,"snaptrim_duration":0,"stat_sum":{"num_bytes":6067459,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":114,"num_read_kb":98570,"num_write":79,"num_write_kb":22078,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"224'274","reported_seq":909,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-24T01:59:03.087853+0000","last_change":"2024-04-24T01:59:00.087226+0000","last_active":"2024-04-24T01:59:03.087853+0000","last_peered":"2024-04-24T01:59:03.087853+0000","last_clean":"2024-04-24T01:59:03.087853+0000","last_became_active":"2024-04-24T01:55:30.321866+0000","last_became_peered":"2024-04-24T01:55:30.321866+0000","last_unstale":"2024-04-24T01:59:03.087853+0000","last_undegraded":"2024-04-24T01:59:03.087853+0000","last_fullsized":"2024-04-24T01:59:03.087853+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":274,"log_dups_size":0,"ondisk_log_size":274,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":22,"snaptrim_duration":0,"stat_sum":{"num_bytes":7227109,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":154,"num_read_kb":106054,"num_write":133,"num_write_kb":36094,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"223'220","reported_seq":865,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-24T01:59:03.145156+0000","last_change":"2024-04-24T01:58:59.080967+0000","last_active":"2024-04-24T01:59:03.145156+0000","last_peered":"2024-04-24T01:59:03.145156+0000","last_clean":"2024-04-24T01:59:03.145156+0000","last_became_active":"2024-04-24T01:55:30.320186+0000","last_became_peered":"2024-04-24T01:55:30.320186+0000","last_unstale":"2024-04-24T01:59:03.145156+0000","last_undegraded":"2024-04-24T01:59:03.145156+0000","last_fullsized":"2024-04-24T01:59:03.145156+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":220,"log_dups_size":0,"ondisk_log_size":220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":22,"snaptrim_duration":0,"stat_sum":{"num_bytes":5949810,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":110,"num_read_kb":71601,"num_write":119,"num_write_kb":33001,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"220'237","reported_seq":892,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-24T01:59:03.113503+0000","last_change":"2024-04-24T01:58:59.116719+0000","last_active":"2024-04-24T01:59:03.113503+0000","last_peered":"2024-04-24T01:59:03.113503+0000","last_clean":"2024-04-24T01:59:03.113503+0000","last_became_active":"2024-04-24T01:55:30.288081+0000","last_became_peered":"2024-04-24T01:55:30.288081+0000","last_unstale":"2024-04-24T01:59:03.113503+0000","last_undegraded":"2024-04-24T01:59:03.113503+0000","last_fullsized":"2024-04-24T01:59:03.113503+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":237,"log_dups_size":0,"ondisk_log_size":237,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":23,"snaptrim_duration":0,"stat_sum":{"num_bytes":9849100,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":128,"num_read_kb":92549,"num_write":110,"num_write_kb":30252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"228'73","reported_seq":733,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-04-24T01:59:04.127583+0000","last_change":"2024-04-24T01:59:04.150263+0000","last_active":"2024-04-24T01:59:04.127583+0000","last_peered":"2024-04-24T01:59:04.127583+0000","last_clean":"2024-04-24T01:59:04.127583+0000","last_became_active":"2024-04-24T01:55:30.333815+0000","last_became_peered":"2024-04-24T01:55:30.333815+0000","last_unstale":"2024-04-24T01:59:04.127583+0000","last_undegraded":"2024-04-24T01:59:04.127583+0000","last_fullsized":"2024-04-24T01:59:04.127583+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0,"stat_sum":{"num_bytes":7512795,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":17558,"num_write":46,"num_write_kb":13299,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"223'445","reported_seq":1056,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-24T01:59:03.112073+0000","last_change":"2024-04-24T01:59:00.089921+0000","last_active":"2024-04-24T01:59:03.112073+0000","last_peered":"2024-04-24T01:59:03.112073+0000","last_clean":"2024-04-24T01:59:03.112073+0000","last_became_active":"2024-04-24T01:55:30.315075+0000","last_became_peered":"2024-04-24T01:55:30.315075+0000","last_unstale":"2024-04-24T01:59:03.112073+0000","last_undegraded":"2024-04-24T01:59:03.112073+0000","last_fullsized":"2024-04-24T01:59:03.112073+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":445,"log_dups_size":0,"ondisk_log_size":445,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":42,"snaptrim_duration":0,"stat_sum":{"num_bytes":6033845,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":203,"num_read_kb":131581,"num_write":191,"num_write_kb":50968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"213'74","reported_seq":742,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-24T01:59:03.128927+0000","last_change":"2024-04-24T01:58:59.135365+0000","last_active":"2024-04-24T01:59:03.128927+0000","last_peered":"2024-04-24T01:59:03.128927+0000","last_clean":"2024-04-24T01:59:03.128927+0000","last_became_active":"2024-04-24T01:55:30.350300+0000","last_became_peered":"2024-04-24T01:55:30.350300+0000","last_unstale":"2024-04-24T01:59:03.128927+0000","last_undegraded":"2024-04-24T01:59:03.128927+0000","last_fullsized":"2024-04-24T01:59:03.128927+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":8,"snaptrim_duration":0,"stat_sum":{"num_bytes":3058940,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":16965,"num_write":47,"num_write_kb":11517,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"225'22","reported_seq":682,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-24T01:59:03.123816+0000","last_change":"2024-04-24T01:58:59.098102+0000","last_active":"2024-04-24T01:59:03.123816+0000","last_peered":"2024-04-24T01:59:03.123816+0000","last_clean":"2024-04-24T01:59:03.123816+0000","last_became_active":"2024-04-24T01:55:30.376813+0000","last_became_peered":"2024-04-24T01:55:30.376813+0000","last_unstale":"2024-04-24T01:59:03.123816+0000","last_undegraded":"2024-04-24T01:59:03.123816+0000","last_fullsized":"2024-04-24T01:59:03.123816+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3798448,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":4,"num_write":21,"num_write_kb":5614,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'97","reported_seq":702,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-04-24T01:59:04.105305+0000","last_change":"2024-04-24T01:55:16.196470+0000","last_active":"2024-04-24T01:59:04.105305+0000","last_peered":"2024-04-24T01:59:04.105305+0000","last_clean":"2024-04-24T01:59:04.105305+0000","last_became_active":"2024-04-24T01:55:16.190464+0000","last_became_peered":"2024-04-24T01:55:16.190464+0000","last_unstale":"2024-04-24T01:59:04.105305+0000","last_undegraded":"2024-04-24T01:59:04.105305+0000","last_fullsized":"2024-04-24T01:59:04.105305+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:15.127115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:15.127115+0000","last_clean_scrub_stamp":"2024-04-24T01:55:15.127115+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":568,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-24T01:59:03.124002+0000","last_change":"2024-04-24T01:55:18.236148+0000","last_active":"2024-04-24T01:59:03.124002+0000","last_peered":"2024-04-24T01:59:03.124002+0000","last_clean":"2024-04-24T01:59:03.124002+0000","last_became_active":"2024-04-24T01:55:18.230154+0000","last_became_peered":"2024-04-24T01:55:18.230154+0000","last_unstale":"2024-04-24T01:59:03.124002+0000","last_undegraded":"2024-04-24T01:59:03.124002+0000","last_fullsized":"2024-04-24T01:59:03.124002+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"224'344","reported_seq":989,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-24T01:59:03.124943+0000","last_change":"2024-04-24T01:58:59.099143+0000","last_active":"2024-04-24T01:59:03.124943+0000","last_peered":"2024-04-24T01:59:03.124943+0000","last_clean":"2024-04-24T01:59:03.124943+0000","last_became_active":"2024-04-24T01:55:30.316222+0000","last_became_peered":"2024-04-24T01:55:30.316222+0000","last_unstale":"2024-04-24T01:59:03.124943+0000","last_undegraded":"2024-04-24T01:59:03.124943+0000","last_fullsized":"2024-04-24T01:59:03.124943+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":344,"log_dups_size":0,"ondisk_log_size":344,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":27,"snaptrim_duration":0,"stat_sum":{"num_bytes":6554862,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":179,"num_read_kb":119004,"num_write":176,"num_write_kb":48929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":578,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-24T01:59:03.127565+0000","last_change":"2024-04-24T01:55:18.265905+0000","last_active":"2024-04-24T01:59:03.127565+0000","last_peered":"2024-04-24T01:59:03.127565+0000","last_clean":"2024-04-24T01:59:03.127565+0000","last_became_active":"2024-04-24T01:55:18.257020+0000","last_became_peered":"2024-04-24T01:55:18.257020+0000","last_unstale":"2024-04-24T01:59:03.127565+0000","last_undegraded":"2024-04-24T01:59:03.127565+0000","last_fullsized":"2024-04-24T01:59:03.127565+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"228'281","reported_seq":909,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-04-24T01:59:04.118693+0000","last_change":"2024-04-24T01:59:04.145119+0000","last_active":"2024-04-24T01:59:04.118693+0000","last_peered":"2024-04-24T01:59:04.118693+0000","last_clean":"2024-04-24T01:59:04.118693+0000","last_became_active":"2024-04-24T01:55:30.344129+0000","last_became_peered":"2024-04-24T01:55:30.344129+0000","last_unstale":"2024-04-24T01:59:04.118693+0000","last_undegraded":"2024-04-24T01:59:04.118693+0000","last_fullsized":"2024-04-24T01:59:04.118693+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":281,"log_dups_size":0,"ondisk_log_size":281,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":21,"snaptrim_duration":0,"stat_sum":{"num_bytes":3731220,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":124,"num_read_kb":72534,"num_write":125,"num_write_kb":35228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":568,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-24T01:59:03.114572+0000","last_change":"2024-04-24T01:55:18.215251+0000","last_active":"2024-04-24T01:59:03.114572+0000","last_peered":"2024-04-24T01:59:03.114572+0000","last_clean":"2024-04-24T01:59:03.114572+0000","last_became_active":"2024-04-24T01:55:18.206605+0000","last_became_peered":"2024-04-24T01:55:18.206605+0000","last_unstale":"2024-04-24T01:59:03.114572+0000","last_undegraded":"2024-04-24T01:59:03.114572+0000","last_fullsized":"2024-04-24T01:59:03.114572+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"217'138","reported_seq":809,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-04-24T01:59:04.122284+0000","last_change":"2024-04-24T01:59:04.125087+0000","last_active":"2024-04-24T01:59:04.122284+0000","last_peered":"2024-04-24T01:59:04.122284+0000","last_clean":"2024-04-24T01:59:04.122284+0000","last_became_active":"2024-04-24T01:55:30.401366+0000","last_became_peered":"2024-04-24T01:55:30.401366+0000","last_unstale":"2024-04-24T01:59:04.122284+0000","last_undegraded":"2024-04-24T01:59:04.122284+0000","last_fullsized":"2024-04-24T01:59:04.122284+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":10,"snaptrim_duration":0,"stat_sum":{"num_bytes":5529635,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":116,"num_read_kb":89047,"num_write":51,"num_write_kb":13418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":578,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-24T01:59:03.134067+0000","last_change":"2024-04-24T01:55:18.230893+0000","last_active":"2024-04-24T01:59:03.134067+0000","last_peered":"2024-04-24T01:59:03.134067+0000","last_clean":"2024-04-24T01:59:03.134067+0000","last_became_active":"2024-04-24T01:55:18.225188+0000","last_became_peered":"2024-04-24T01:55:18.225188+0000","last_unstale":"2024-04-24T01:59:03.134067+0000","last_undegraded":"2024-04-24T01:59:03.134067+0000","last_fullsized":"2024-04-24T01:59:03.134067+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"227'229","reported_seq":849,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-24T01:59:03.156066+0000","last_change":"2024-04-24T01:58:59.092520+0000","last_active":"2024-04-24T01:59:03.156066+0000","last_peered":"2024-04-24T01:59:03.156066+0000","last_clean":"2024-04-24T01:59:03.156066+0000","last_became_active":"2024-04-24T01:55:30.343318+0000","last_became_peered":"2024-04-24T01:55:30.343318+0000","last_unstale":"2024-04-24T01:59:03.156066+0000","last_undegraded":"2024-04-24T01:59:03.156066+0000","last_fullsized":"2024-04-24T01:59:03.156066+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":229,"log_dups_size":0,"ondisk_log_size":229,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":20,"snaptrim_duration":0,"stat_sum":{"num_bytes":6117880,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":106,"num_read_kb":83023,"num_write":102,"num_write_kb":28542,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"224'304","reported_seq":944,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-24T01:59:03.122004+0000","last_change":"2024-04-24T01:58:59.098890+0000","last_active":"2024-04-24T01:59:03.122004+0000","last_peered":"2024-04-24T01:59:03.122004+0000","last_clean":"2024-04-24T01:59:03.122004+0000","last_became_active":"2024-04-24T01:55:30.345308+0000","last_became_peered":"2024-04-24T01:55:30.345308+0000","last_unstale":"2024-04-24T01:59:03.122004+0000","last_undegraded":"2024-04-24T01:59:03.122004+0000","last_fullsized":"2024-04-24T01:59:03.122004+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":304,"log_dups_size":0,"ondisk_log_size":304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":24,"snaptrim_duration":0,"stat_sum":{"num_bytes":10033977,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":163,"num_read_kb":110063,"num_write":139,"num_write_kb":37031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":554,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-24T01:59:03.095150+0000","last_change":"2024-04-24T01:55:18.224238+0000","last_active":"2024-04-24T01:59:03.095150+0000","last_peered":"2024-04-24T01:59:03.095150+0000","last_clean":"2024-04-24T01:59:03.095150+0000","last_became_active":"2024-04-24T01:55:18.219131+0000","last_became_peered":"2024-04-24T01:55:18.219131+0000","last_unstale":"2024-04-24T01:59:03.095150+0000","last_undegraded":"2024-04-24T01:59:03.095150+0000","last_fullsized":"2024-04-24T01:59:03.095150+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"217'58","reported_seq":705,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-24T01:59:03.100166+0000","last_change":"2024-04-24T01:58:59.101427+0000","last_active":"2024-04-24T01:59:03.100166+0000","last_peered":"2024-04-24T01:59:03.100166+0000","last_clean":"2024-04-24T01:59:03.100166+0000","last_became_active":"2024-04-24T01:55:30.319958+0000","last_became_peered":"2024-04-24T01:55:30.319958+0000","last_unstale":"2024-04-24T01:59:03.100166+0000","last_undegraded":"2024-04-24T01:59:03.100166+0000","last_fullsized":"2024-04-24T01:59:03.100166+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":5,"snaptrim_duration":0,"stat_sum":{"num_bytes":3428694,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":27,"num_read_kb":19609,"num_write":42,"num_write_kb":11889,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":569,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-04-24T01:59:04.120808+0000","last_change":"2024-04-24T01:55:18.235763+0000","last_active":"2024-04-24T01:59:04.120808+0000","last_peered":"2024-04-24T01:59:04.120808+0000","last_clean":"2024-04-24T01:59:04.120808+0000","last_became_active":"2024-04-24T01:55:18.229175+0000","last_became_peered":"2024-04-24T01:55:18.229175+0000","last_unstale":"2024-04-24T01:59:04.120808+0000","last_undegraded":"2024-04-24T01:59:04.120808+0000","last_fullsized":"2024-04-24T01:59:04.120808+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"223'102","reported_seq":752,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-04-24T01:59:04.098291+0000","last_change":"2024-04-24T01:59:04.114513+0000","last_active":"2024-04-24T01:59:04.098291+0000","last_peered":"2024-04-24T01:59:04.098291+0000","last_clean":"2024-04-24T01:59:04.098291+0000","last_became_active":"2024-04-24T01:55:30.372690+0000","last_became_peered":"2024-04-24T01:55:30.372690+0000","last_unstale":"2024-04-24T01:59:04.098291+0000","last_undegraded":"2024-04-24T01:59:04.098291+0000","last_fullsized":"2024-04-24T01:59:04.098291+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":9,"snaptrim_duration":0,"stat_sum":{"num_bytes":3630378,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":48,"num_read_kb":30460,"num_write":45,"num_write_kb":12824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":554,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-24T01:59:03.097919+0000","last_change":"2024-04-24T01:55:18.278351+0000","last_active":"2024-04-24T01:59:03.097919+0000","last_peered":"2024-04-24T01:59:03.097919+0000","last_clean":"2024-04-24T01:59:03.097919+0000","last_became_active":"2024-04-24T01:55:18.273668+0000","last_became_peered":"2024-04-24T01:55:18.273668+0000","last_unstale":"2024-04-24T01:59:03.097919+0000","last_undegraded":"2024-04-24T01:59:03.097919+0000","last_fullsized":"2024-04-24T01:59:03.097919+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"224'231","reported_seq":886,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-24T01:59:03.102123+0000","last_change":"2024-04-24T01:58:59.102023+0000","last_active":"2024-04-24T01:59:03.102123+0000","last_peered":"2024-04-24T01:59:03.102123+0000","last_clean":"2024-04-24T01:59:03.102123+0000","last_became_active":"2024-04-24T01:55:30.363123+0000","last_became_peered":"2024-04-24T01:55:30.363123+0000","last_unstale":"2024-04-24T01:59:03.102123+0000","last_undegraded":"2024-04-24T01:59:03.102123+0000","last_fullsized":"2024-04-24T01:59:03.102123+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":231,"log_dups_size":0,"ondisk_log_size":231,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":23,"snaptrim_duration":0,"stat_sum":{"num_bytes":5983391,"num_objects":6,"num_object_clones":2,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":159,"num_read_kb":124304,"num_write":128,"num_write_kb":33878,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":568,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-24T01:59:03.126603+0000","last_change":"2024-04-24T01:55:18.241117+0000","last_active":"2024-04-24T01:59:03.126603+0000","last_peered":"2024-04-24T01:59:03.126603+0000","last_clean":"2024-04-24T01:59:03.126603+0000","last_became_active":"2024-04-24T01:55:18.234313+0000","last_became_peered":"2024-04-24T01:55:18.234313+0000","last_unstale":"2024-04-24T01:59:03.126603+0000","last_undegraded":"2024-04-24T01:59:03.126603+0000","last_fullsized":"2024-04-24T01:59:03.126603+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":94507543,"num_objects":37,"num_object_clones":5,"num_object_copies":111,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":1708,"num_read_kb":1182926,"num_write":1554,"num_write_kb":424562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":193236992,"data_stored":197821646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3198,"ondisk_log_size":3198,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":1}],"osd_stats":[{"osd":3,"up_from":10,"seq":42949673189,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":97588,"kb_used_data":62444,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94274252,"statfs":{"total":96636764160,"available":96536834048,"internally_reserved":0,"allocated":63942656,"data_stored":65313449,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673189,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":82536,"kb_used_data":51552,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94289304,"statfs":{"total":96636764160,"available":96552247296,"internally_reserved":0,"allocated":52789248,"data_stored":53962608,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705894,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":73472,"kb_used_data":38328,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94298368,"statfs":{"total":96636764160,"available":96561528832,"internally_reserved":0,"allocated":39247872,"data_stored":38793202,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705893,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":85072,"kb_used_data":49928,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94286768,"statfs":{"total":96636764160,"available":96549650432,"internally_reserved":0,"allocated":51126272,"data_stored":51955914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":34836480,"data_stored":34494986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49270784,"data_stored":51150004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49590272,"data_stored":51157818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":59539456,"data_stored":61018838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T01:59:08.807 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T01:59:08.914 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:59:08.913+0000 7f6a5066d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:59:08.918 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:59:08.918+0000 7f6a5066d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:59:09.050 INFO:tasks.rados.rados.0.smithi079.stdout:951: done (0 left) 2024-04-24T01:59:09.050 INFO:tasks.rados.rados.0.smithi079.stdout:952: write oid 13 current snap is 107 2024-04-24T01:59:09.050 INFO:tasks.rados.rados.0.smithi079.stdout:952: seq_num 470 ranges {699290=464563,1603478=535155,2823983=765846} 2024-04-24T01:59:09.056 INFO:tasks.rados.rados.0.smithi079.stdout:952: writing smithi07929762-13 from 699290 to 1163853 tid 1 2024-04-24T01:59:09.063 INFO:tasks.rados.rados.0.smithi079.stdout:952: writing smithi07929762-13 from 1603478 to 2138633 tid 2 2024-04-24T01:59:09.072 INFO:tasks.rados.rados.0.smithi079.stdout:952: writing smithi07929762-13 from 2823983 to 3589829 tid 3 2024-04-24T01:59:09.072 INFO:tasks.rados.rados.0.smithi079.stdout:953: write (excl) oid 4 current snap is 107 2024-04-24T01:59:09.072 INFO:tasks.rados.rados.0.smithi079.stdout:953: seq_num 471 ranges {716097=539812,1945607=795166,3535053=381044} 2024-04-24T01:59:09.079 INFO:tasks.rados.rados.0.smithi079.stdout:953: writing smithi07929762-4 from 716097 to 1255909 tid 1 2024-04-24T01:59:09.090 INFO:tasks.rados.rados.0.smithi079.stdout:953: writing smithi07929762-4 from 1945607 to 2740773 tid 2 2024-04-24T01:59:09.098 INFO:tasks.rados.rados.0.smithi079.stdout:953: writing smithi07929762-4 from 3535053 to 3916097 tid 3 2024-04-24T01:59:09.098 INFO:tasks.rados.rados.0.smithi079.stdout:954: read oid 27 snap -1 2024-04-24T01:59:09.098 INFO:tasks.rados.rados.0.smithi079.stdout:954: expect (ObjNum 459 snap 103 seq_num 459) 2024-04-24T01:59:09.098 INFO:tasks.rados.rados.0.smithi079.stdout:955: read oid 16 snap -1 2024-04-24T01:59:09.098 INFO:tasks.rados.rados.0.smithi079.stdout:955: expect deleted 2024-04-24T01:59:09.098 INFO:tasks.rados.rados.0.smithi079.stdout:956: read oid 49 snap -1 2024-04-24T01:59:09.098 INFO:tasks.rados.rados.0.smithi079.stdout:956: expect deleted 2024-04-24T01:59:09.098 INFO:tasks.rados.rados.0.smithi079.stdout:957: delete oid 47 current snap is 107 2024-04-24T01:59:09.155 INFO:tasks.rados.rados.0.smithi079.stdout:953: finishing write tid 1 to smithi07929762-4 2024-04-24T01:59:09.157 INFO:tasks.rados.rados.0.smithi079.stdout:952: finishing write tid 1 to smithi07929762-13 2024-04-24T01:59:09.180 INFO:tasks.rados.rados.0.smithi079.stdout:953: finishing write tid 2 to smithi07929762-4 2024-04-24T01:59:09.183 INFO:tasks.rados.rados.0.smithi079.stdout:952: finishing write tid 2 to smithi07929762-13 2024-04-24T01:59:09.192 INFO:tasks.rados.rados.0.smithi079.stdout:953: finishing write tid 3 to smithi07929762-4 2024-04-24T01:59:09.200 INFO:tasks.rados.rados.0.smithi079.stdout:953: finishing write tid 4 to smithi07929762-4 2024-04-24T01:59:09.203 INFO:tasks.rados.rados.0.smithi079.stdout:953: finishing write tid 5 to smithi07929762-4 2024-04-24T01:59:09.203 INFO:tasks.rados.rados.0.smithi079.stdout:953: oid 4 updating version 0 to 78 2024-04-24T01:59:09.203 INFO:tasks.rados.rados.0.smithi079.stdout:953: oid 4 version 78 is already newer than 75 2024-04-24T01:59:09.203 INFO:tasks.rados.rados.0.smithi079.stdout:953: oid 4 version 78 is already newer than 78 2024-04-24T01:59:09.203 INFO:tasks.rados.rados.0.smithi079.stdout:953: oid 4 version 78 is already newer than 78 2024-04-24T01:59:09.203 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 4 v 78 (ObjNum 471 snap 107 seq_num 471) dirty exists 2024-04-24T01:59:09.203 INFO:tasks.rados.rados.0.smithi079.stdout:953: left oid 4 (ObjNum 471 snap 107 seq_num 471) 2024-04-24T01:59:09.205 INFO:tasks.rados.rados.0.smithi079.stdout:952: finishing write tid 3 to smithi07929762-13 2024-04-24T01:59:09.212 INFO:tasks.rados.rados.0.smithi079.stdout:954: expect (ObjNum 459 snap 103 seq_num 459) 2024-04-24T01:59:09.406 INFO:tasks.rados.rados.0.smithi079.stdout:952: finishing write tid 4 to smithi07929762-13 2024-04-24T01:59:09.407 INFO:tasks.rados.rados.0.smithi079.stdout:952: finishing write tid 5 to smithi07929762-13 2024-04-24T01:59:09.407 INFO:tasks.rados.rados.0.smithi079.stdout:952: oid 13 updating version 0 to 348 2024-04-24T01:59:09.407 INFO:tasks.rados.rados.0.smithi079.stdout:952: oid 13 version 348 is already newer than 348 2024-04-24T01:59:09.407 INFO:tasks.rados.rados.0.smithi079.stdout:952: oid 13 version 348 is already newer than 348 2024-04-24T01:59:09.407 INFO:tasks.rados.rados.0.smithi079.stdout:952: oid 13 version 348 is already newer than 348 2024-04-24T01:59:09.407 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 13 v 348 (ObjNum 470 snap 107 seq_num 470) dirty exists 2024-04-24T01:59:09.407 INFO:tasks.rados.rados.0.smithi079.stdout:952: left oid 13 (ObjNum 470 snap 107 seq_num 470) 2024-04-24T01:59:09.407 INFO:tasks.rados.rados.0.smithi079.stdout:952: done (5 left) 2024-04-24T01:59:09.407 INFO:tasks.rados.rados.0.smithi079.stdout:953: done (4 left) 2024-04-24T01:59:09.407 INFO:tasks.rados.rados.0.smithi079.stdout:954: done (3 left) 2024-04-24T01:59:09.407 INFO:tasks.rados.rados.0.smithi079.stdout:955: done (2 left) 2024-04-24T01:59:09.407 INFO:tasks.rados.rados.0.smithi079.stdout:956: done (1 left) 2024-04-24T01:59:09.407 INFO:tasks.rados.rados.0.smithi079.stdout:957: done (0 left) 2024-04-24T01:59:09.407 INFO:tasks.rados.rados.0.smithi079.stdout:958: snap_create 2024-04-24T01:59:10.056 INFO:tasks.rados.rados.0.smithi079.stdout:958: done (0 left) 2024-04-24T01:59:10.056 INFO:tasks.rados.rados.0.smithi079.stdout:959: snap_create 2024-04-24T01:59:11.012 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T01:59:11.063 INFO:tasks.rados.rados.0.smithi079.stdout:959: done (0 left) 2024-04-24T01:59:11.063 INFO:tasks.rados.rados.0.smithi079.stdout:960: write (excl) oid 44 current snap is 109 2024-04-24T01:59:11.063 INFO:tasks.rados.rados.0.smithi079.stdout:960: seq_num 473 ranges {749711=406663,1778924=682482,3189899=716015} 2024-04-24T01:59:11.071 INFO:tasks.rados.rados.0.smithi079.stdout:960: writing smithi07929762-44 from 749711 to 1156374 tid 1 2024-04-24T01:59:11.080 INFO:tasks.rados.rados.0.smithi079.stdout:960: writing smithi07929762-44 from 1778924 to 2461406 tid 2 2024-04-24T01:59:11.089 INFO:tasks.rados.rados.0.smithi079.stdout:960: writing smithi07929762-44 from 3189899 to 3905914 tid 3 2024-04-24T01:59:11.089 INFO:tasks.rados.rados.0.smithi079.stdout:961: delete oid 38 current snap is 109 2024-04-24T01:59:11.121 INFO:tasks.rados.rados.0.smithi079.stdout:961: done (1 left) 2024-04-24T01:59:11.121 INFO:tasks.rados.rados.0.smithi079.stdout:962: write (excl) oid 13 current snap is 109 2024-04-24T01:59:11.121 INFO:tasks.rados.rados.0.smithi079.stdout:962: seq_num 475 ranges {783325=557161,1779535=686151,3244745=454912} 2024-04-24T01:59:11.128 INFO:tasks.rados.rados.0.smithi079.stdout:962: writing smithi07929762-13 from 783325 to 1340486 tid 1 2024-04-24T01:59:11.132 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:59:11.131+0000 7f75d1d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:59:11.135 INFO:tasks.rados.rados.0.smithi079.stdout:962: writing smithi07929762-13 from 1779535 to 2465686 tid 2 2024-04-24T01:59:11.139 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:59:11.139+0000 7f75d1d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:59:11.144 INFO:tasks.rados.rados.0.smithi079.stdout:962: writing smithi07929762-13 from 3244745 to 3699657 tid 3 2024-04-24T01:59:11.144 INFO:tasks.rados.rados.0.smithi079.stdout:963: delete oid 17 current snap is 109 2024-04-24T01:59:11.146 INFO:tasks.rados.rados.0.smithi079.stdout:960: finishing write tid 1 to smithi07929762-44 2024-04-24T01:59:11.164 INFO:tasks.rados.rados.0.smithi079.stdout:963: done (2 left) 2024-04-24T01:59:11.164 INFO:tasks.rados.rados.0.smithi079.stdout:964: write oid 18 current snap is 109 2024-04-24T01:59:11.165 INFO:tasks.rados.rados.0.smithi079.stdout:964: seq_num 477 ranges {416939=707659,1896499=120440} 2024-04-24T01:59:11.170 INFO:tasks.rados.rados.0.smithi079.stdout:964: writing smithi07929762-18 from 416939 to 1124598 tid 1 2024-04-24T01:59:11.176 INFO:tasks.rados.rados.0.smithi079.stdout:964: writing smithi07929762-18 from 1896499 to 2016939 tid 2 2024-04-24T01:59:11.176 INFO:tasks.rados.rados.0.smithi079.stdout:965: read oid 42 snap -1 2024-04-24T01:59:11.176 INFO:tasks.rados.rados.0.smithi079.stdout:965: expect (ObjNum 434 snap 99 seq_num 434) 2024-04-24T01:59:11.176 INFO:tasks.rados.rados.0.smithi079.stdout:966: write (excl) oid 24 current snap is 109 2024-04-24T01:59:11.176 INFO:tasks.rados.rados.0.smithi079.stdout:966: seq_num 478 ranges {433746=782908,1954981=78765} 2024-04-24T01:59:11.186 INFO:tasks.rados.rados.0.smithi079.stdout:966: writing smithi07929762-24 from 433746 to 1216654 tid 1 2024-04-24T01:59:11.191 INFO:tasks.rados.rados.0.smithi079.stdout:966: writing smithi07929762-24 from 1954981 to 2033746 tid 2 2024-04-24T01:59:11.191 INFO:tasks.rados.rados.0.smithi079.stdout:967: write oid 27 current snap is 109 2024-04-24T01:59:11.191 INFO:tasks.rados.rados.0.smithi079.stdout:960: finishing write tid 2 to smithi07929762-44 2024-04-24T01:59:11.191 INFO:tasks.rados.rados.0.smithi079.stdout:967: seq_num 479 ranges {450553=574510,1613463=437090} 2024-04-24T01:59:11.198 INFO:tasks.rados.rados.0.smithi079.stdout:967: writing smithi07929762-27 from 450553 to 1025063 tid 1 2024-04-24T01:59:11.204 INFO:tasks.rados.rados.0.smithi079.stdout:967: writing smithi07929762-27 from 1613463 to 2050553 tid 2 2024-04-24T01:59:11.205 INFO:tasks.rados.rados.0.smithi079.stdout:968: snap_create 2024-04-24T01:59:11.205 INFO:tasks.rados.rados.0.smithi079.stdout:962: finishing write tid 1 to smithi07929762-13 2024-04-24T01:59:11.208 INFO:tasks.rados.rados.0.smithi079.stdout:960: finishing write tid 3 to smithi07929762-44 2024-04-24T01:59:11.224 INFO:tasks.rados.rados.0.smithi079.stdout:965: expect (ObjNum 434 snap 99 seq_num 434) 2024-04-24T01:59:11.447 INFO:tasks.rados.rados.0.smithi079.stdout:960: finishing write tid 4 to smithi07929762-44 2024-04-24T01:59:11.447 INFO:tasks.rados.rados.0.smithi079.stdout:960: finishing write tid 5 to smithi07929762-44 2024-04-24T01:59:11.447 INFO:tasks.rados.rados.0.smithi079.stdout:960: oid 44 updating version 0 to 234 2024-04-24T01:59:11.447 INFO:tasks.rados.rados.0.smithi079.stdout:960: oid 44 version 234 is already newer than 231 2024-04-24T01:59:11.448 INFO:tasks.rados.rados.0.smithi079.stdout:960: oid 44 version 234 is already newer than 234 2024-04-24T01:59:11.448 INFO:tasks.rados.rados.0.smithi079.stdout:960: oid 44 version 234 is already newer than 234 2024-04-24T01:59:11.448 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 44 v 234 (ObjNum 473 snap 109 seq_num 473) dirty exists 2024-04-24T01:59:11.448 INFO:tasks.rados.rados.0.smithi079.stdout:960: left oid 44 (ObjNum 473 snap 109 seq_num 473) 2024-04-24T01:59:11.448 INFO:tasks.rados.rados.0.smithi079.stdout:962: finishing write tid 2 to smithi07929762-13 2024-04-24T01:59:11.448 INFO:tasks.rados.rados.0.smithi079.stdout:964: finishing write tid 1 to smithi07929762-18 2024-04-24T01:59:11.448 INFO:tasks.rados.rados.0.smithi079.stdout:964: finishing write tid 2 to smithi07929762-18 2024-04-24T01:59:11.448 INFO:tasks.rados.rados.0.smithi079.stdout:962: finishing write tid 3 to smithi07929762-13 2024-04-24T01:59:11.448 INFO:tasks.rados.rados.0.smithi079.stdout:962: finishing write tid 4 to smithi07929762-13 2024-04-24T01:59:11.448 INFO:tasks.rados.rados.0.smithi079.stdout:962: finishing write tid 5 to smithi07929762-13 2024-04-24T01:59:11.448 INFO:tasks.rados.rados.0.smithi079.stdout:962: oid 13 updating version 0 to 351 2024-04-24T01:59:11.448 INFO:tasks.rados.rados.0.smithi079.stdout:962: oid 13 updating version 351 to 354 2024-04-24T01:59:11.448 INFO:tasks.rados.rados.0.smithi079.stdout:962: oid 13 version 354 is already newer than 354 2024-04-24T01:59:11.448 INFO:tasks.rados.rados.0.smithi079.stdout:962: oid 13 version 354 is already newer than 354 2024-04-24T01:59:11.448 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 13 v 354 (ObjNum 475 snap 109 seq_num 475) dirty exists 2024-04-24T01:59:11.449 INFO:tasks.rados.rados.0.smithi079.stdout:962: left oid 13 (ObjNum 475 snap 109 seq_num 475) 2024-04-24T01:59:11.449 INFO:tasks.rados.rados.0.smithi079.stdout:966: finishing write tid 1 to smithi07929762-24 2024-04-24T01:59:11.449 INFO:tasks.rados.rados.0.smithi079.stdout:964: finishing write tid 3 to smithi07929762-18 2024-04-24T01:59:11.449 INFO:tasks.rados.rados.0.smithi079.stdout:964: finishing write tid 4 to smithi07929762-18 2024-04-24T01:59:11.449 INFO:tasks.rados.rados.0.smithi079.stdout:964: oid 18 updating version 0 to 448 2024-04-24T01:59:11.449 INFO:tasks.rados.rados.0.smithi079.stdout:964: oid 18 version 448 is already newer than 448 2024-04-24T01:59:11.449 INFO:tasks.rados.rados.0.smithi079.stdout:964: oid 18 version 448 is already newer than 448 2024-04-24T01:59:11.449 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 18 v 448 (ObjNum 477 snap 109 seq_num 477) dirty exists 2024-04-24T01:59:11.449 INFO:tasks.rados.rados.0.smithi079.stdout:964: left oid 18 (ObjNum 477 snap 109 seq_num 477) 2024-04-24T01:59:11.449 INFO:tasks.rados.rados.0.smithi079.stdout:967: finishing write tid 1 to smithi07929762-27 2024-04-24T01:59:11.449 INFO:tasks.rados.rados.0.smithi079.stdout:967: finishing write tid 2 to smithi07929762-27 2024-04-24T01:59:11.450 INFO:tasks.rados.rados.0.smithi079.stdout:966: finishing write tid 2 to smithi07929762-24 2024-04-24T01:59:11.450 INFO:tasks.rados.rados.0.smithi079.stdout:967: finishing write tid 3 to smithi07929762-27 2024-04-24T01:59:11.450 INFO:tasks.rados.rados.0.smithi079.stdout:967: finishing write tid 4 to smithi07929762-27 2024-04-24T01:59:11.450 INFO:tasks.rados.rados.0.smithi079.stdout:967: oid 27 updating version 0 to 77 2024-04-24T01:59:11.450 INFO:tasks.rados.rados.0.smithi079.stdout:967: oid 27 version 77 is already newer than 77 2024-04-24T01:59:11.450 INFO:tasks.rados.rados.0.smithi079.stdout:967: oid 27 version 77 is already newer than 77 2024-04-24T01:59:11.450 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 27 v 77 (ObjNum 479 snap 109 seq_num 479) dirty exists 2024-04-24T01:59:11.450 INFO:tasks.rados.rados.0.smithi079.stdout:967: left oid 27 (ObjNum 479 snap 109 seq_num 479) 2024-04-24T01:59:11.450 INFO:tasks.rados.rados.0.smithi079.stdout:966: finishing write tid 3 to smithi07929762-24 2024-04-24T01:59:11.450 INFO:tasks.rados.rados.0.smithi079.stdout:966: finishing write tid 4 to smithi07929762-24 2024-04-24T01:59:11.450 INFO:tasks.rados.rados.0.smithi079.stdout:966: oid 24 updating version 0 to 297 2024-04-24T01:59:11.450 INFO:tasks.rados.rados.0.smithi079.stdout:966: oid 24 version 297 is already newer than 297 2024-04-24T01:59:11.450 INFO:tasks.rados.rados.0.smithi079.stdout:966: oid 24 version 297 is already newer than 295 2024-04-24T01:59:11.450 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 24 v 297 (ObjNum 478 snap 109 seq_num 478) dirty exists 2024-04-24T01:59:11.451 INFO:tasks.rados.rados.0.smithi079.stdout:966: left oid 24 (ObjNum 478 snap 109 seq_num 478) 2024-04-24T01:59:12.070 INFO:tasks.rados.rados.0.smithi079.stdout:960: done (6 left) 2024-04-24T01:59:12.070 INFO:tasks.rados.rados.0.smithi079.stdout:962: done (5 left) 2024-04-24T01:59:12.070 INFO:tasks.rados.rados.0.smithi079.stdout:964: done (4 left) 2024-04-24T01:59:12.070 INFO:tasks.rados.rados.0.smithi079.stdout:965: done (3 left) 2024-04-24T01:59:12.070 INFO:tasks.rados.rados.0.smithi079.stdout:966: done (2 left) 2024-04-24T01:59:12.070 INFO:tasks.rados.rados.0.smithi079.stdout:967: done (1 left) 2024-04-24T01:59:12.070 INFO:tasks.rados.rados.0.smithi079.stdout:968: done (0 left) 2024-04-24T01:59:12.070 INFO:tasks.rados.rados.0.smithi079.stdout:969: write oid 46 current snap is 110 2024-04-24T01:59:12.070 INFO:tasks.rados.rados.0.smithi079.stdout:969: seq_num 480 ranges {467360=649759,1671945=395415} 2024-04-24T01:59:12.076 INFO:tasks.rados.rados.0.smithi079.stdout:969: writing smithi07929762-46 from 467360 to 1117119 tid 1 2024-04-24T01:59:12.081 INFO:tasks.rados.rados.0.smithi079.stdout:969: writing smithi07929762-46 from 1671945 to 2067360 tid 2 2024-04-24T01:59:12.081 INFO:tasks.rados.rados.0.smithi079.stdout:970: read oid 45 snap 108 2024-04-24T01:59:12.081 INFO:tasks.rados.rados.0.smithi079.stdout:970: expect deleted 2024-04-24T01:59:12.081 INFO:tasks.rados.rados.0.smithi079.stdout:971: snap_create 2024-04-24T01:59:12.147 INFO:tasks.rados.rados.0.smithi079.stdout:969: finishing write tid 1 to smithi07929762-46 2024-04-24T01:59:12.159 INFO:tasks.rados.rados.0.smithi079.stdout:969: finishing write tid 2 to smithi07929762-46 2024-04-24T01:59:12.166 INFO:tasks.rados.rados.0.smithi079.stdout:969: finishing write tid 3 to smithi07929762-46 2024-04-24T01:59:12.171 INFO:tasks.rados.rados.0.smithi079.stdout:969: finishing write tid 4 to smithi07929762-46 2024-04-24T01:59:12.171 INFO:tasks.rados.rados.0.smithi079.stdout:969: oid 46 updating version 0 to 245 2024-04-24T01:59:12.171 INFO:tasks.rados.rados.0.smithi079.stdout:969: oid 46 version 245 is already newer than 245 2024-04-24T01:59:12.171 INFO:tasks.rados.rados.0.smithi079.stdout:969: oid 46 version 245 is already newer than 245 2024-04-24T01:59:12.171 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 46 v 245 (ObjNum 480 snap 110 seq_num 480) dirty exists 2024-04-24T01:59:12.171 INFO:tasks.rados.rados.0.smithi079.stdout:969: left oid 46 (ObjNum 480 snap 110 seq_num 480) 2024-04-24T01:59:13.072 INFO:tasks.rados.rados.0.smithi079.stdout:969: done (2 left) 2024-04-24T01:59:13.073 INFO:tasks.rados.rados.0.smithi079.stdout:970: done (1 left) 2024-04-24T01:59:13.073 INFO:tasks.rados.rados.0.smithi079.stdout:971: done (0 left) 2024-04-24T01:59:13.073 INFO:tasks.rados.rados.0.smithi079.stdout:972: read oid 26 snap 108 2024-04-24T01:59:13.073 INFO:tasks.rados.rados.0.smithi079.stdout:972: expect (ObjNum 458 snap 103 seq_num 458) 2024-04-24T01:59:13.073 INFO:tasks.rados.rados.0.smithi079.stdout:973: snap_remove snap 110 2024-04-24T01:59:13.351 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-04-24T01:59:13.351 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-04-24T01:59:13.351 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-04-24T01:59:13.351 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-04-24T01:59:13.367 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T01:59:13.463 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:59:13.457+0000 7f411ee6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:59:13.467 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:59:13.467+0000 7f411ee6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:59:13.482 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T01:59:13.482 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T01:59:13.482 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T01:59:13.482 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T01:59:13.482 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T01:59:13.482 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T01:59:13.482 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T01:59:13.482 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T01:59:13.482 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T01:59:13.483 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T01:59:13.483 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T01:59:13.483 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T01:59:13.485 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T01:59:13.485 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T01:59:13.485 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-04-24T01:59:13.503 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:59:13.502+0000 7f06f786d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:59:13.508 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:59:13.508+0000 7f06f786d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:59:13.600 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:59:13.594+0000 7f9244aad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:59:13.603 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:59:13.604+0000 7f9244aad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:59:13.618 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T01:59:13.618 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T01:59:13.618 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T01:59:13.618 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T01:59:13.618 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T01:59:13.619 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T01:59:13.619 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T01:59:13.619 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T01:59:13.619 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T01:59:13.619 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T01:59:13.619 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T01:59:13.619 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T01:59:13.621 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T01:59:13.621 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T01:59:13.621 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-04-24T01:59:13.769 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:59:13.765+0000 7f5ef3a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:59:13.773 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:59:13.773+0000 7f5ef3a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:59:13.790 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T01:59:13.790 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T01:59:13.790 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T01:59:13.790 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T01:59:13.790 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T01:59:13.790 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T01:59:13.790 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T01:59:13.790 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T01:59:13.790 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T01:59:13.790 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T01:59:13.790 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T01:59:13.791 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T01:59:13.793 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T01:59:13.793 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T01:59:13.793 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-04-24T01:59:13.901 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:59:13.897+0000 7ff43366d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:59:13.905 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:59:13.905+0000 7ff43366d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:59:13.922 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T01:59:13.922 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T01:59:13.922 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T01:59:13.922 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T01:59:13.922 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T01:59:13.922 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T01:59:13.922 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T01:59:13.923 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T01:59:13.923 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T01:59:13.923 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T01:59:13.923 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T01:59:13.923 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T01:59:13.925 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T01:59:13.925 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T01:59:14.073 INFO:tasks.rados.rados.0.smithi079.stdout:973: done (1 left) 2024-04-24T01:59:14.073 INFO:tasks.rados.rados.0.smithi079.stdout:974: delete oid 28 current snap is 111 2024-04-24T01:59:14.073 INFO:tasks.rados.rados.0.smithi079.stdout:972: expect (ObjNum 458 snap 103 seq_num 458) 2024-04-24T01:59:14.217 INFO:tasks.rados.rados.0.smithi079.stdout:972: done (1 left) 2024-04-24T01:59:14.217 INFO:tasks.rados.rados.0.smithi079.stdout:974: done (0 left) 2024-04-24T01:59:14.217 INFO:tasks.rados.rados.0.smithi079.stdout:975: snap_remove snap 109 2024-04-24T01:59:15.092 INFO:tasks.rados.rados.0.smithi079.stdout:975: done (0 left) 2024-04-24T01:59:15.092 INFO:tasks.rados.rados.0.smithi079.stdout:976: delete oid 19 current snap is 111 2024-04-24T01:59:15.132 INFO:tasks.rados.rados.0.smithi079.stdout:976: done (0 left) 2024-04-24T01:59:15.132 INFO:tasks.rados.rados.0.smithi079.stdout:977: snap_remove snap 108 2024-04-24T01:59:15.609 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T01:59:15.717 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:59:15.716+0000 7f46c7606640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:59:15.721 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:59:15.721+0000 7f46c7606640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:59:16.046 INFO:tasks.rados.rados.0.smithi079.stdout:977: done (0 left) 2024-04-24T01:59:16.046 INFO:tasks.rados.rados.0.smithi079.stdout:978: snap_remove snap 107 2024-04-24T01:59:17.049 INFO:tasks.rados.rados.0.smithi079.stdout:978: done (0 left) 2024-04-24T01:59:17.049 INFO:tasks.rados.rados.0.smithi079.stdout:979: delete oid 46 current snap is 111 2024-04-24T01:59:17.137 INFO:tasks.rados.rados.0.smithi079.stdout:979: done (0 left) 2024-04-24T01:59:17.137 INFO:tasks.rados.rados.0.smithi079.stdout:980: write (excl) oid 37 current snap is 111 2024-04-24T01:59:17.137 INFO:tasks.rados.rados.0.smithi079.stdout:980: seq_num 484 ranges {534588=550755,1789520=345068} 2024-04-24T01:59:17.142 INFO:tasks.rados.rados.0.smithi079.stdout:980: writing smithi07929762-37 from 534588 to 1085343 tid 1 2024-04-24T01:59:17.150 INFO:tasks.rados.rados.0.smithi079.stdout:980: writing smithi07929762-37 from 1789520 to 2134588 tid 2 2024-04-24T01:59:17.150 INFO:tasks.rados.rados.0.smithi079.stdout:981: read oid 29 snap -1 2024-04-24T01:59:17.150 INFO:tasks.rados.rados.0.smithi079.stdout:981: expect deleted 2024-04-24T01:59:17.150 INFO:tasks.rados.rados.0.smithi079.stdout:982: read oid 10 snap -1 2024-04-24T01:59:17.151 INFO:tasks.rados.rados.0.smithi079.stdout:982: expect (ObjNum 338 snap 81 seq_num 338) 2024-04-24T01:59:17.151 INFO:tasks.rados.rados.0.smithi079.stdout:983: read oid 14 snap -1 2024-04-24T01:59:17.151 INFO:tasks.rados.rados.0.smithi079.stdout:983: expect deleted 2024-04-24T01:59:17.151 INFO:tasks.rados.rados.0.smithi079.stdout:984: write (excl) oid 34 current snap is 111 2024-04-24T01:59:17.151 INFO:tasks.rados.rados.0.smithi079.stdout:984: seq_num 485 ranges {551395=626004,1848002=303393} 2024-04-24T01:59:17.158 INFO:tasks.rados.rados.0.smithi079.stdout:984: writing smithi07929762-34 from 551395 to 1177399 tid 1 2024-04-24T01:59:17.164 INFO:tasks.rados.rados.0.smithi079.stdout:984: writing smithi07929762-34 from 1848002 to 2151395 tid 2 2024-04-24T01:59:17.164 INFO:tasks.rados.rados.0.smithi079.stdout:985: read oid 47 snap -1 2024-04-24T01:59:17.164 INFO:tasks.rados.rados.0.smithi079.stdout:985: expect deleted 2024-04-24T01:59:17.164 INFO:tasks.rados.rados.0.smithi079.stdout:986: write (excl) oid 20 current snap is 111 2024-04-24T01:59:17.164 INFO:tasks.rados.rados.0.smithi079.stdout:986: seq_num 486 ranges {568202=701253,1906484=261718} 2024-04-24T01:59:17.171 INFO:tasks.rados.rados.0.smithi079.stdout:986: writing smithi07929762-20 from 568202 to 1269455 tid 1 2024-04-24T01:59:17.177 INFO:tasks.rados.rados.0.smithi079.stdout:986: writing smithi07929762-20 from 1906484 to 2168202 tid 2 2024-04-24T01:59:17.177 INFO:tasks.rados.rados.0.smithi079.stdout:987: read oid 8 snap -1 2024-04-24T01:59:17.177 INFO:tasks.rados.rados.0.smithi079.stdout:987: expect (ObjNum 464 snap 104 seq_num 464) 2024-04-24T01:59:17.177 INFO:tasks.rados.rados.0.smithi079.stdout:988: write oid 45 current snap is 111 2024-04-24T01:59:17.177 INFO:tasks.rados.rados.0.smithi079.stdout:988: seq_num 487 ranges {585009=492855,1564966=620043} 2024-04-24T01:59:17.183 INFO:tasks.rados.rados.0.smithi079.stdout:988: writing smithi07929762-45 from 585009 to 1077864 tid 1 2024-04-24T01:59:17.190 INFO:tasks.rados.rados.0.smithi079.stdout:988: writing smithi07929762-45 from 1564966 to 2185009 tid 2 2024-04-24T01:59:17.190 INFO:tasks.rados.rados.0.smithi079.stdout:989: delete oid 21 current snap is 111 2024-04-24T01:59:17.190 INFO:tasks.rados.rados.0.smithi079.stdout:982: expect (ObjNum 338 snap 81 seq_num 338) 2024-04-24T01:59:17.408 INFO:tasks.rados.rados.0.smithi079.stdout:982: done (9 left) 2024-04-24T01:59:17.409 INFO:tasks.rados.rados.0.smithi079.stdout:989: done (8 left) 2024-04-24T01:59:17.409 INFO:tasks.rados.rados.0.smithi079.stdout:990: write oid 9 current snap is 111 2024-04-24T01:59:17.409 INFO:tasks.rados.rados.0.smithi079.stdout:980: finishing write tid 1 to smithi07929762-37 2024-04-24T01:59:17.409 INFO:tasks.rados.rados.0.smithi079.stdout:984: finishing write tid 1 to smithi07929762-34 2024-04-24T01:59:17.409 INFO:tasks.rados.rados.0.smithi079.stdout:988: finishing write tid 1 to smithi07929762-45 2024-04-24T01:59:17.409 INFO:tasks.rados.rados.0.smithi079.stdout:990: seq_num 489 ranges {618623=643353,1681930=536693} 2024-04-24T01:59:17.415 INFO:tasks.rados.rados.0.smithi079.stdout:990: writing smithi07929762-9 from 618623 to 1261976 tid 1 2024-04-24T01:59:17.421 INFO:tasks.rados.rados.0.smithi079.stdout:990: writing smithi07929762-9 from 1681930 to 2218623 tid 2 2024-04-24T01:59:17.421 INFO:tasks.rados.rados.0.smithi079.stdout:985: done (8 left) 2024-04-24T01:59:17.422 INFO:tasks.rados.rados.0.smithi079.stdout:991: delete oid 13 current snap is 111 2024-04-24T01:59:17.422 INFO:tasks.rados.rados.0.smithi079.stdout:984: finishing write tid 2 to smithi07929762-34 2024-04-24T01:59:17.422 INFO:tasks.rados.rados.0.smithi079.stdout:984: finishing write tid 3 to smithi07929762-34 2024-04-24T01:59:17.422 INFO:tasks.rados.rados.0.smithi079.stdout:988: finishing write tid 2 to smithi07929762-45 2024-04-24T01:59:17.422 INFO:tasks.rados.rados.0.smithi079.stdout:984: finishing write tid 4 to smithi07929762-34 2024-04-24T01:59:17.422 INFO:tasks.rados.rados.0.smithi079.stdout:984: oid 34 updating version 0 to 253 2024-04-24T01:59:17.422 INFO:tasks.rados.rados.0.smithi079.stdout:984: oid 34 version 253 is already newer than 251 2024-04-24T01:59:17.422 INFO:tasks.rados.rados.0.smithi079.stdout:984: oid 34 version 253 is already newer than 253 2024-04-24T01:59:17.422 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 34 v 253 (ObjNum 485 snap 111 seq_num 485) dirty exists 2024-04-24T01:59:17.422 INFO:tasks.rados.rados.0.smithi079.stdout:984: left oid 34 (ObjNum 485 snap 111 seq_num 485) 2024-04-24T01:59:17.422 INFO:tasks.rados.rados.0.smithi079.stdout:980: finishing write tid 2 to smithi07929762-37 2024-04-24T01:59:17.422 INFO:tasks.rados.rados.0.smithi079.stdout:988: finishing write tid 3 to smithi07929762-45 2024-04-24T01:59:17.423 INFO:tasks.rados.rados.0.smithi079.stdout:988: finishing write tid 4 to smithi07929762-45 2024-04-24T01:59:17.423 INFO:tasks.rados.rados.0.smithi079.stdout:988: oid 45 updating version 0 to 285 2024-04-24T01:59:17.423 INFO:tasks.rados.rados.0.smithi079.stdout:988: oid 45 version 285 is already newer than 285 2024-04-24T01:59:17.423 INFO:tasks.rados.rados.0.smithi079.stdout:988: oid 45 version 285 is already newer than 285 2024-04-24T01:59:17.423 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 45 v 285 (ObjNum 487 snap 111 seq_num 487) dirty exists 2024-04-24T01:59:17.423 INFO:tasks.rados.rados.0.smithi079.stdout:988: left oid 45 (ObjNum 487 snap 111 seq_num 487) 2024-04-24T01:59:17.423 INFO:tasks.rados.rados.0.smithi079.stdout:980: finishing write tid 3 to smithi07929762-37 2024-04-24T01:59:17.423 INFO:tasks.rados.rados.0.smithi079.stdout:980: finishing write tid 4 to smithi07929762-37 2024-04-24T01:59:17.423 INFO:tasks.rados.rados.0.smithi079.stdout:980: oid 37 updating version 0 to 451 2024-04-24T01:59:17.423 INFO:tasks.rados.rados.0.smithi079.stdout:980: oid 37 version 451 is already newer than 449 2024-04-24T01:59:17.423 INFO:tasks.rados.rados.0.smithi079.stdout:980: oid 37 version 451 is already newer than 451 2024-04-24T01:59:17.423 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 37 v 451 (ObjNum 484 snap 111 seq_num 484) dirty exists 2024-04-24T01:59:17.423 INFO:tasks.rados.rados.0.smithi079.stdout:980: left oid 37 (ObjNum 484 snap 111 seq_num 484) 2024-04-24T01:59:17.423 INFO:tasks.rados.rados.0.smithi079.stdout:986: finishing write tid 1 to smithi07929762-20 2024-04-24T01:59:17.424 INFO:tasks.rados.rados.0.smithi079.stdout:986: finishing write tid 2 to smithi07929762-20 2024-04-24T01:59:17.424 INFO:tasks.rados.rados.0.smithi079.stdout:987: expect (ObjNum 464 snap 104 seq_num 464) 2024-04-24T01:59:17.683 INFO:tasks.rados.rados.0.smithi079.stdout:986: finishing write tid 3 to smithi07929762-20 2024-04-24T01:59:17.683 INFO:tasks.rados.rados.0.smithi079.stdout:986: finishing write tid 4 to smithi07929762-20 2024-04-24T01:59:17.683 INFO:tasks.rados.rados.0.smithi079.stdout:986: oid 20 updating version 0 to 307 2024-04-24T01:59:17.683 INFO:tasks.rados.rados.0.smithi079.stdout:986: oid 20 updating version 307 to 309 2024-04-24T01:59:17.683 INFO:tasks.rados.rados.0.smithi079.stdout:986: oid 20 version 309 is already newer than 309 2024-04-24T01:59:17.683 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 20 v 309 (ObjNum 486 snap 111 seq_num 486) dirty exists 2024-04-24T01:59:17.683 INFO:tasks.rados.rados.0.smithi079.stdout:986: left oid 20 (ObjNum 486 snap 111 seq_num 486) 2024-04-24T01:59:17.684 INFO:tasks.rados.rados.0.smithi079.stdout:980: done (8 left) 2024-04-24T01:59:17.684 INFO:tasks.rados.rados.0.smithi079.stdout:981: done (7 left) 2024-04-24T01:59:17.684 INFO:tasks.rados.rados.0.smithi079.stdout:983: done (6 left) 2024-04-24T01:59:17.684 INFO:tasks.rados.rados.0.smithi079.stdout:984: done (5 left) 2024-04-24T01:59:17.684 INFO:tasks.rados.rados.0.smithi079.stdout:986: done (4 left) 2024-04-24T01:59:17.684 INFO:tasks.rados.rados.0.smithi079.stdout:987: done (3 left) 2024-04-24T01:59:17.684 INFO:tasks.rados.rados.0.smithi079.stdout:988: done (2 left) 2024-04-24T01:59:17.684 INFO:tasks.rados.rados.0.smithi079.stdout:991: done (1 left) 2024-04-24T01:59:17.684 INFO:tasks.rados.rados.0.smithi079.stdout:992: snap_create 2024-04-24T01:59:17.684 INFO:tasks.rados.rados.0.smithi079.stdout:990: finishing write tid 1 to smithi07929762-9 2024-04-24T01:59:17.684 INFO:tasks.rados.rados.0.smithi079.stdout:990: finishing write tid 2 to smithi07929762-9 2024-04-24T01:59:17.684 INFO:tasks.rados.rados.0.smithi079.stdout:990: finishing write tid 3 to smithi07929762-9 2024-04-24T01:59:17.684 INFO:tasks.rados.rados.0.smithi079.stdout:990: finishing write tid 4 to smithi07929762-9 2024-04-24T01:59:17.684 INFO:tasks.rados.rados.0.smithi079.stdout:990: oid 9 updating version 0 to 312 2024-04-24T01:59:17.684 INFO:tasks.rados.rados.0.smithi079.stdout:990: oid 9 version 312 is already newer than 312 2024-04-24T01:59:17.684 INFO:tasks.rados.rados.0.smithi079.stdout:990: oid 9 version 312 is already newer than 312 2024-04-24T01:59:17.685 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 9 v 312 (ObjNum 489 snap 111 seq_num 489) dirty exists 2024-04-24T01:59:17.685 INFO:tasks.rados.rados.0.smithi079.stdout:990: left oid 9 (ObjNum 489 snap 111 seq_num 489) 2024-04-24T01:59:17.818 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T01:59:17.923 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:59:17.921+0000 7f3d6866d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:59:17.927 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:59:17.926+0000 7f3d6866d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:59:18.093 INFO:tasks.rados.rados.0.smithi079.stdout:990: done (1 left) 2024-04-24T01:59:18.093 INFO:tasks.rados.rados.0.smithi079.stdout:992: done (0 left) 2024-04-24T01:59:18.093 INFO:tasks.rados.rados.0.smithi079.stdout:993: write (excl) oid 41 current snap is 112 2024-04-24T01:59:18.093 INFO:tasks.rados.rados.0.smithi079.stdout:993: seq_num 491 ranges {652237=793851,2198894=53343} 2024-04-24T01:59:18.102 INFO:tasks.rados.rados.0.smithi079.stdout:993: writing smithi07929762-41 from 652237 to 1446088 tid 1 2024-04-24T01:59:18.107 INFO:tasks.rados.rados.0.smithi079.stdout:993: writing smithi07929762-41 from 2198894 to 2252237 tid 2 2024-04-24T01:59:18.107 INFO:tasks.rados.rados.0.smithi079.stdout:994: read oid 17 snap -1 2024-04-24T01:59:18.108 INFO:tasks.rados.rados.0.smithi079.stdout:994: expect deleted 2024-04-24T01:59:18.108 INFO:tasks.rados.rados.0.smithi079.stdout:995: read oid 39 snap -1 2024-04-24T01:59:18.108 INFO:tasks.rados.rados.0.smithi079.stdout:995: expect deleted 2024-04-24T01:59:18.108 INFO:tasks.rados.rados.0.smithi079.stdout:996: read oid 18 snap -1 2024-04-24T01:59:18.108 INFO:tasks.rados.rados.0.smithi079.stdout:996: expect (ObjNum 477 snap 109 seq_num 477) 2024-04-24T01:59:18.108 INFO:tasks.rados.rados.0.smithi079.stdout:997: read oid 6 snap 111 2024-04-24T01:59:18.108 INFO:tasks.rados.rados.0.smithi079.stdout:997: expect (ObjNum 276 snap 71 seq_num 276) 2024-04-24T01:59:18.108 INFO:tasks.rados.rados.0.smithi079.stdout:998: read oid 30 snap 111 2024-04-24T01:59:18.109 INFO:tasks.rados.rados.0.smithi079.stdout:998: expect (ObjNum 454 snap 102 seq_num 454) 2024-04-24T01:59:18.109 INFO:tasks.rados.rados.0.smithi079.stdout:999: read oid 1 snap -1 2024-04-24T01:59:18.109 INFO:tasks.rados.rados.0.smithi079.stdout:999: expect (ObjNum 413 snap 97 seq_num 413) 2024-04-24T01:59:18.109 INFO:tasks.rados.rados.0.smithi079.stdout:1000: read oid 23 snap -1 2024-04-24T01:59:18.109 INFO:tasks.rados.rados.0.smithi079.stdout:1000: expect deleted 2024-04-24T01:59:18.109 INFO:tasks.rados.rados.0.smithi079.stdout:1001: write (excl) oid 34 current snap is 112 2024-04-24T01:59:18.109 INFO:tasks.rados.rados.0.smithi079.stdout:1001: seq_num 492 ranges {669044=469100,1741023=459161} 2024-04-24T01:59:18.115 INFO:tasks.rados.rados.0.smithi079.stdout:1001: writing smithi07929762-34 from 669044 to 1138144 tid 1 2024-04-24T01:59:18.123 INFO:tasks.rados.rados.0.smithi079.stdout:1001: writing smithi07929762-34 from 1741023 to 2200184 tid 2 2024-04-24T01:59:18.124 INFO:tasks.rados.rados.0.smithi079.stdout:1002: write oid 45 current snap is 112 2024-04-24T01:59:18.124 INFO:tasks.rados.rados.0.smithi079.stdout:1002: seq_num 493 ranges {685851=544349,1799505=486346} 2024-04-24T01:59:18.133 INFO:tasks.rados.rados.0.smithi079.stdout:1002: writing smithi07929762-45 from 685851 to 1230200 tid 1 2024-04-24T01:59:18.140 INFO:tasks.rados.rados.0.smithi079.stdout:1002: writing smithi07929762-45 from 1799505 to 2285851 tid 2 2024-04-24T01:59:18.140 INFO:tasks.rados.rados.0.smithi079.stdout:1003: read oid 4 snap -1 2024-04-24T01:59:18.140 INFO:tasks.rados.rados.0.smithi079.stdout:1003: expect (ObjNum 471 snap 107 seq_num 471) 2024-04-24T01:59:18.140 INFO:tasks.rados.rados.0.smithi079.stdout:1004: read oid 44 snap -1 2024-04-24T01:59:18.140 INFO:tasks.rados.rados.0.smithi079.stdout:1004: expect (ObjNum 473 snap 109 seq_num 473) 2024-04-24T01:59:18.140 INFO:tasks.rados.rados.0.smithi079.stdout:1005: delete oid 36 current snap is 112 2024-04-24T01:59:18.185 INFO:tasks.rados.rados.0.smithi079.stdout:1003: expect (ObjNum 471 snap 107 seq_num 471) 2024-04-24T01:59:18.398 INFO:tasks.rados.rados.0.smithi079.stdout:996: expect (ObjNum 477 snap 109 seq_num 477) 2024-04-24T01:59:18.474 INFO:tasks.rados.rados.0.smithi079.stdout:1001: finishing write tid 1 to smithi07929762-34 2024-04-24T01:59:18.474 INFO:tasks.rados.rados.0.smithi079.stdout:994: done (12 left) 2024-04-24T01:59:18.475 INFO:tasks.rados.rados.0.smithi079.stdout:995: done (11 left) 2024-04-24T01:59:18.475 INFO:tasks.rados.rados.0.smithi079.stdout:996: done (10 left) 2024-04-24T01:59:18.475 INFO:tasks.rados.rados.0.smithi079.stdout:1000: done (9 left) 2024-04-24T01:59:18.475 INFO:tasks.rados.rados.0.smithi079.stdout:1003: done (8 left) 2024-04-24T01:59:18.475 INFO:tasks.rados.rados.0.smithi079.stdout:1005: done (7 left) 2024-04-24T01:59:18.476 INFO:tasks.rados.rados.0.smithi079.stdout:1006: write (excl) oid 27 current snap is 112 2024-04-24T01:59:18.476 INFO:tasks.rados.rados.0.smithi079.stdout:1004: expect (ObjNum 473 snap 109 seq_num 473) 2024-04-24T01:59:18.680 INFO:tasks.rados.rados.0.smithi079.stdout:1002: finishing write tid 1 to smithi07929762-45 2024-04-24T01:59:18.680 INFO:tasks.rados.rados.0.smithi079.stdout:999: expect (ObjNum 413 snap 97 seq_num 413) 2024-04-24T01:59:18.828 INFO:tasks.rados.rados.0.smithi079.stdout:998: expect (ObjNum 454 snap 102 seq_num 454) 2024-04-24T01:59:18.926 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-04-24T01:59:18.926 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-04-24T01:59:18.926 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-04-24T01:59:18.926 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-04-24T01:59:18.964 INFO:tasks.rados.rados.0.smithi079.stdout:997: expect (ObjNum 276 snap 71 seq_num 276) 2024-04-24T01:59:19.036 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:59:19.030+0000 7f827682d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:59:19.040 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:59:19.039+0000 7f827682d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:59:19.054 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T01:59:19.055 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T01:59:19.055 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T01:59:19.055 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T01:59:19.055 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T01:59:19.055 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T01:59:19.055 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T01:59:19.055 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T01:59:19.055 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T01:59:19.055 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T01:59:19.055 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T01:59:19.055 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T01:59:19.058 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T01:59:19.059 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T01:59:19.059 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-04-24T01:59:19.190 INFO:tasks.rados.rados.0.smithi079.stdout:1002: finishing write tid 2 to smithi07929762-45 2024-04-24T01:59:19.190 INFO:tasks.rados.rados.0.smithi079.stdout:1002: finishing write tid 3 to smithi07929762-45 2024-04-24T01:59:19.190 INFO:tasks.rados.rados.0.smithi079.stdout:993: finishing write tid 1 to smithi07929762-41 2024-04-24T01:59:19.190 INFO:tasks.rados.rados.0.smithi079.stdout:1002: finishing write tid 4 to smithi07929762-45 2024-04-24T01:59:19.190 INFO:tasks.rados.rados.0.smithi079.stdout:1002: oid 45 updating version 0 to 290 2024-04-24T01:59:19.190 INFO:tasks.rados.rados.0.smithi079.stdout:1002: oid 45 version 290 is already newer than 290 2024-04-24T01:59:19.190 INFO:tasks.rados.rados.0.smithi079.stdout:1002: oid 45 version 290 is already newer than 290 2024-04-24T01:59:19.191 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 45 v 290 (ObjNum 493 snap 112 seq_num 493) dirty exists 2024-04-24T01:59:19.191 INFO:tasks.rados.rados.0.smithi079.stdout:1002: left oid 45 (ObjNum 493 snap 112 seq_num 493) 2024-04-24T01:59:19.191 INFO:tasks.rados.rados.0.smithi079.stdout:993: finishing write tid 2 to smithi07929762-41 2024-04-24T01:59:19.191 INFO:tasks.rados.rados.0.smithi079.stdout:1006: seq_num 495 ranges {719465=411200,1632822=686643} 2024-04-24T01:59:19.196 INFO:tasks.rados.rados.0.smithi079.stdout:1006: writing smithi07929762-27 from 719465 to 1130665 tid 1 2024-04-24T01:59:19.203 INFO:tasks.rados.rados.0.smithi079.stdout:1006: writing smithi07929762-27 from 1632822 to 2319465 tid 2 2024-04-24T01:59:19.204 INFO:tasks.rados.rados.0.smithi079.stdout:997: done (7 left) 2024-04-24T01:59:19.204 INFO:tasks.rados.rados.0.smithi079.stdout:998: done (6 left) 2024-04-24T01:59:19.204 INFO:tasks.rados.rados.0.smithi079.stdout:999: done (5 left) 2024-04-24T01:59:19.204 INFO:tasks.rados.rados.0.smithi079.stdout:1002: done (4 left) 2024-04-24T01:59:19.204 INFO:tasks.rados.rados.0.smithi079.stdout:1004: done (3 left) 2024-04-24T01:59:19.204 INFO:tasks.rados.rados.0.smithi079.stdout:1007: read oid 48 snap 111 2024-04-24T01:59:19.204 INFO:tasks.rados.rados.0.smithi079.stdout:1007: expect (ObjNum 468 snap 107 seq_num 468) 2024-04-24T01:59:19.204 INFO:tasks.rados.rados.0.smithi079.stdout:1008: delete oid 3 current snap is 112 2024-04-24T01:59:19.204 INFO:tasks.rados.rados.0.smithi079.stdout:993: finishing write tid 3 to smithi07929762-41 2024-04-24T01:59:19.204 INFO:tasks.rados.rados.0.smithi079.stdout:993: finishing write tid 4 to smithi07929762-41 2024-04-24T01:59:19.204 INFO:tasks.rados.rados.0.smithi079.stdout:993: oid 41 updating version 0 to 142 2024-04-24T01:59:19.205 INFO:tasks.rados.rados.0.smithi079.stdout:993: oid 41 version 142 is already newer than 142 2024-04-24T01:59:19.205 INFO:tasks.rados.rados.0.smithi079.stdout:993: oid 41 version 142 is already newer than 140 2024-04-24T01:59:19.205 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 41 v 142 (ObjNum 491 snap 112 seq_num 491) dirty exists 2024-04-24T01:59:19.205 INFO:tasks.rados.rados.0.smithi079.stdout:993: left oid 41 (ObjNum 491 snap 112 seq_num 491) 2024-04-24T01:59:19.205 INFO:tasks.rados.rados.0.smithi079.stdout:1001: finishing write tid 2 to smithi07929762-34 2024-04-24T01:59:19.205 INFO:tasks.rados.rados.0.smithi079.stdout:1001: finishing write tid 3 to smithi07929762-34 2024-04-24T01:59:19.205 INFO:tasks.rados.rados.0.smithi079.stdout:1001: finishing write tid 4 to smithi07929762-34 2024-04-24T01:59:19.205 INFO:tasks.rados.rados.0.smithi079.stdout:1001: oid 34 updating version 0 to 257 2024-04-24T01:59:19.205 INFO:tasks.rados.rados.0.smithi079.stdout:1001: oid 34 version 257 is already newer than 257 2024-04-24T01:59:19.205 INFO:tasks.rados.rados.0.smithi079.stdout:1001: oid 34 version 257 is already newer than 255 2024-04-24T01:59:19.206 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 34 v 257 (ObjNum 492 snap 112 seq_num 492) dirty exists 2024-04-24T01:59:19.206 INFO:tasks.rados.rados.0.smithi079.stdout:1001: left oid 34 (ObjNum 492 snap 112 seq_num 492) 2024-04-24T01:59:19.215 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:59:19.208+0000 7f5cc995d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:59:19.220 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:59:19.219+0000 7f5cc995d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:59:19.240 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T01:59:19.240 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T01:59:19.240 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T01:59:19.240 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T01:59:19.240 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T01:59:19.240 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T01:59:19.240 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T01:59:19.240 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T01:59:19.241 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T01:59:19.241 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T01:59:19.241 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T01:59:19.241 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T01:59:19.245 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T01:59:19.245 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T01:59:19.245 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-04-24T01:59:19.256 INFO:tasks.rados.rados.0.smithi079.stdout:993: done (4 left) 2024-04-24T01:59:19.256 INFO:tasks.rados.rados.0.smithi079.stdout:1001: done (3 left) 2024-04-24T01:59:19.256 INFO:tasks.rados.rados.0.smithi079.stdout:1008: done (2 left) 2024-04-24T01:59:19.256 INFO:tasks.rados.rados.0.smithi079.stdout:1009: write (excl) oid 46 current snap is 112 2024-04-24T01:59:19.256 INFO:tasks.rados.rados.0.smithi079.stdout:1009: seq_num 497 ranges {753079=561698,2033433=319646} 2024-04-24T01:59:19.263 INFO:tasks.rados.rados.0.smithi079.stdout:1009: writing smithi07929762-46 from 753079 to 1314777 tid 1 2024-04-24T01:59:19.271 INFO:tasks.rados.rados.0.smithi079.stdout:1009: writing smithi07929762-46 from 2033433 to 2353079 tid 2 2024-04-24T01:59:19.271 INFO:tasks.rados.rados.0.smithi079.stdout:1010: read oid 26 snap -1 2024-04-24T01:59:19.271 INFO:tasks.rados.rados.0.smithi079.stdout:1010: expect (ObjNum 458 snap 103 seq_num 458) 2024-04-24T01:59:19.271 INFO:tasks.rados.rados.0.smithi079.stdout:1011: write oid 3 current snap is 112 2024-04-24T01:59:19.271 INFO:tasks.rados.rados.0.smithi079.stdout:1006: finishing write tid 1 to smithi07929762-27 2024-04-24T01:59:19.271 INFO:tasks.rados.rados.0.smithi079.stdout:1011: seq_num 498 ranges {769886=636947,2091915=277971} 2024-04-24T01:59:19.280 INFO:tasks.rados.rados.0.smithi079.stdout:1011: writing smithi07929762-3 from 769886 to 1406833 tid 1 2024-04-24T01:59:19.287 INFO:tasks.rados.rados.0.smithi079.stdout:1011: writing smithi07929762-3 from 2091915 to 2369886 tid 2 2024-04-24T01:59:19.287 INFO:tasks.rados.rados.0.smithi079.stdout:1012: write oid 49 current snap is 112 2024-04-24T01:59:19.287 INFO:tasks.rados.rados.0.smithi079.stdout:1012: seq_num 499 ranges {786693=712196,2150397=236296} 2024-04-24T01:59:19.298 INFO:tasks.rados.rados.0.smithi079.stdout:1012: writing smithi07929762-49 from 786693 to 1498889 tid 1 2024-04-24T01:59:19.303 INFO:tasks.rados.rados.0.smithi079.stdout:1012: writing smithi07929762-49 from 2150397 to 2386693 tid 2 2024-04-24T01:59:19.303 INFO:tasks.rados.rados.0.smithi079.stdout:1013: write oid 21 current snap is 112 2024-04-24T01:59:19.303 INFO:tasks.rados.rados.0.smithi079.stdout:1007: expect (ObjNum 468 snap 107 seq_num 468) 2024-04-24T01:59:19.403 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:59:19.393+0000 7f15e719d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:59:19.406 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:59:19.406+0000 7f15e719d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:59:19.422 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T01:59:19.422 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T01:59:19.422 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T01:59:19.423 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T01:59:19.423 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T01:59:19.423 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T01:59:19.423 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T01:59:19.423 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T01:59:19.423 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T01:59:19.423 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T01:59:19.423 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T01:59:19.423 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T01:59:19.425 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T01:59:19.426 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T01:59:19.426 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-04-24T01:59:19.505 INFO:tasks.rados.rados.0.smithi079.stdout:1010: expect (ObjNum 458 snap 103 seq_num 458) 2024-04-24T01:59:19.575 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:59:19.569+0000 7f0eed42d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:59:19.579 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:59:19.578+0000 7f0eed42d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:59:19.602 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T01:59:19.602 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T01:59:19.602 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T01:59:19.602 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T01:59:19.602 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T01:59:19.602 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T01:59:19.602 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T01:59:19.602 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T01:59:19.602 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T01:59:19.602 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T01:59:19.603 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T01:59:19.603 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T01:59:19.606 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T01:59:19.606 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T01:59:19.652 INFO:tasks.rados.rados.0.smithi079.stdout:1006: finishing write tid 2 to smithi07929762-27 2024-04-24T01:59:19.652 INFO:tasks.rados.rados.0.smithi079.stdout:1006: finishing write tid 3 to smithi07929762-27 2024-04-24T01:59:19.652 INFO:tasks.rados.rados.0.smithi079.stdout:1006: finishing write tid 4 to smithi07929762-27 2024-04-24T01:59:19.652 INFO:tasks.rados.rados.0.smithi079.stdout:1006: oid 27 updating version 0 to 87 2024-04-24T01:59:19.652 INFO:tasks.rados.rados.0.smithi079.stdout:1006: oid 27 version 87 is already newer than 85 2024-04-24T01:59:19.652 INFO:tasks.rados.rados.0.smithi079.stdout:1006: oid 27 version 87 is already newer than 87 2024-04-24T01:59:19.652 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 27 v 87 (ObjNum 495 snap 112 seq_num 495) dirty exists 2024-04-24T01:59:19.652 INFO:tasks.rados.rados.0.smithi079.stdout:1006: left oid 27 (ObjNum 495 snap 112 seq_num 495) 2024-04-24T01:59:19.652 INFO:tasks.rados.rados.0.smithi079.stdout:1009: finishing write tid 1 to smithi07929762-46 2024-04-24T01:59:19.652 INFO:tasks.rados.rados.0.smithi079.stdout:1013: seq_num 500 ranges {403500=787445,1692526=473837} 2024-04-24T01:59:19.658 INFO:tasks.rados.rados.0.smithi079.stdout:1013: writing smithi07929762-21 from 403500 to 1190945 tid 1 2024-04-24T01:59:19.664 INFO:tasks.rados.rados.0.smithi079.stdout:1013: writing smithi07929762-21 from 1692526 to 2166363 tid 2 2024-04-24T01:59:19.665 INFO:tasks.rados.rados.0.smithi079.stdout:1006: done (6 left) 2024-04-24T01:59:19.665 INFO:tasks.rados.rados.0.smithi079.stdout:1007: done (5 left) 2024-04-24T01:59:19.665 INFO:tasks.rados.rados.0.smithi079.stdout:1010: done (4 left) 2024-04-24T01:59:19.665 INFO:tasks.rados.rados.0.smithi079.stdout:1014: delete oid 33 current snap is 112 2024-04-24T01:59:19.665 INFO:tasks.rados.rados.0.smithi079.stdout:1011: finishing write tid 1 to smithi07929762-3 2024-04-24T01:59:19.665 INFO:tasks.rados.rados.0.smithi079.stdout:1009: finishing write tid 2 to smithi07929762-46 2024-04-24T01:59:19.665 INFO:tasks.rados.rados.0.smithi079.stdout:1009: finishing write tid 3 to smithi07929762-46 2024-04-24T01:59:19.665 INFO:tasks.rados.rados.0.smithi079.stdout:1009: finishing write tid 4 to smithi07929762-46 2024-04-24T01:59:19.665 INFO:tasks.rados.rados.0.smithi079.stdout:1009: oid 46 updating version 0 to 260 2024-04-24T01:59:19.665 INFO:tasks.rados.rados.0.smithi079.stdout:1009: oid 46 version 260 is already newer than 260 2024-04-24T01:59:19.665 INFO:tasks.rados.rados.0.smithi079.stdout:1009: oid 46 version 260 is already newer than 258 2024-04-24T01:59:19.665 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 46 v 260 (ObjNum 497 snap 112 seq_num 497) dirty exists 2024-04-24T01:59:19.665 INFO:tasks.rados.rados.0.smithi079.stdout:1009: left oid 46 (ObjNum 497 snap 112 seq_num 497) 2024-04-24T01:59:19.665 INFO:tasks.rados.rados.0.smithi079.stdout:1011: finishing write tid 2 to smithi07929762-3 2024-04-24T01:59:19.665 INFO:tasks.rados.rados.0.smithi079.stdout:1011: finishing write tid 3 to smithi07929762-3 2024-04-24T01:59:19.665 INFO:tasks.rados.rados.0.smithi079.stdout:1011: finishing write tid 4 to smithi07929762-3 2024-04-24T01:59:19.666 INFO:tasks.rados.rados.0.smithi079.stdout:1011: oid 3 updating version 0 to 456 2024-04-24T01:59:19.666 INFO:tasks.rados.rados.0.smithi079.stdout:1011: oid 3 version 456 is already newer than 456 2024-04-24T01:59:19.666 INFO:tasks.rados.rados.0.smithi079.stdout:1011: oid 3 version 456 is already newer than 456 2024-04-24T01:59:19.666 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 3 v 456 (ObjNum 498 snap 112 seq_num 498) dirty exists 2024-04-24T01:59:19.666 INFO:tasks.rados.rados.0.smithi079.stdout:1011: left oid 3 (ObjNum 498 snap 112 seq_num 498) 2024-04-24T01:59:19.666 INFO:tasks.rados.rados.0.smithi079.stdout:1012: finishing write tid 1 to smithi07929762-49 2024-04-24T01:59:19.666 INFO:tasks.rados.rados.0.smithi079.stdout:1012: finishing write tid 2 to smithi07929762-49 2024-04-24T01:59:19.666 INFO:tasks.rados.rados.0.smithi079.stdout:1012: finishing write tid 3 to smithi07929762-49 2024-04-24T01:59:19.666 INFO:tasks.rados.rados.0.smithi079.stdout:1012: finishing write tid 4 to smithi07929762-49 2024-04-24T01:59:19.666 INFO:tasks.rados.rados.0.smithi079.stdout:1012: oid 49 updating version 0 to 459 2024-04-24T01:59:19.666 INFO:tasks.rados.rados.0.smithi079.stdout:1012: oid 49 version 459 is already newer than 459 2024-04-24T01:59:19.666 INFO:tasks.rados.rados.0.smithi079.stdout:1012: oid 49 version 459 is already newer than 459 2024-04-24T01:59:19.666 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 49 v 459 (ObjNum 499 snap 112 seq_num 499) dirty exists 2024-04-24T01:59:19.666 INFO:tasks.rados.rados.0.smithi079.stdout:1012: left oid 49 (ObjNum 499 snap 112 seq_num 499) 2024-04-24T01:59:19.696 INFO:tasks.rados.rados.0.smithi079.stdout:1013: finishing write tid 1 to smithi07929762-21 2024-04-24T01:59:19.706 INFO:tasks.rados.rados.0.smithi079.stdout:1013: finishing write tid 2 to smithi07929762-21 2024-04-24T01:59:19.709 INFO:tasks.rados.rados.0.smithi079.stdout:1013: finishing write tid 3 to smithi07929762-21 2024-04-24T01:59:19.710 INFO:tasks.rados.rados.0.smithi079.stdout:1013: finishing write tid 4 to smithi07929762-21 2024-04-24T01:59:19.711 INFO:tasks.rados.rados.0.smithi079.stdout:1013: oid 21 updating version 0 to 293 2024-04-24T01:59:19.711 INFO:tasks.rados.rados.0.smithi079.stdout:1013: oid 21 version 293 is already newer than 293 2024-04-24T01:59:19.711 INFO:tasks.rados.rados.0.smithi079.stdout:1013: oid 21 version 293 is already newer than 293 2024-04-24T01:59:19.711 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 21 v 293 (ObjNum 500 snap 112 seq_num 500) dirty exists 2024-04-24T01:59:19.711 INFO:tasks.rados.rados.0.smithi079.stdout:1013: left oid 21 (ObjNum 500 snap 112 seq_num 500) 2024-04-24T01:59:19.715 INFO:tasks.rados.rados.0.smithi079.stdout:1009: done (4 left) 2024-04-24T01:59:19.715 INFO:tasks.rados.rados.0.smithi079.stdout:1011: done (3 left) 2024-04-24T01:59:19.715 INFO:tasks.rados.rados.0.smithi079.stdout:1012: done (2 left) 2024-04-24T01:59:19.715 INFO:tasks.rados.rados.0.smithi079.stdout:1013: done (1 left) 2024-04-24T01:59:19.715 INFO:tasks.rados.rados.0.smithi079.stdout:1014: done (0 left) 2024-04-24T01:59:19.715 INFO:tasks.rados.rados.0.smithi079.stdout:1015: delete oid 24 current snap is 112 2024-04-24T01:59:19.737 INFO:tasks.rados.rados.0.smithi079.stdout:1015: done (0 left) 2024-04-24T01:59:19.737 INFO:tasks.rados.rados.0.smithi079.stdout:1016: delete oid 46 current snap is 112 2024-04-24T01:59:19.751 INFO:tasks.rados.rados.0.smithi079.stdout:1016: done (0 left) 2024-04-24T01:59:19.752 INFO:tasks.rados.rados.0.smithi079.stdout:1017: delete oid 47 current snap is 112 2024-04-24T01:59:19.760 INFO:tasks.rados.rados.0.smithi079.stdout:1017: done (0 left) 2024-04-24T01:59:19.760 INFO:tasks.rados.rados.0.smithi079.stdout:1018: delete oid 44 current snap is 112 2024-04-24T01:59:19.792 INFO:tasks.rados.rados.0.smithi079.stdout:1018: done (0 left) 2024-04-24T01:59:19.792 INFO:tasks.rados.rados.0.smithi079.stdout:1019: write (excl) oid 14 current snap is 112 2024-04-24T01:59:19.792 INFO:tasks.rados.rados.0.smithi079.stdout:1019: seq_num 506 ranges {504342=555292,1643418=484844} 2024-04-24T01:59:19.797 INFO:tasks.rados.rados.0.smithi079.stdout:1019: writing smithi07929762-14 from 504342 to 1059634 tid 1 2024-04-24T01:59:19.803 INFO:tasks.rados.rados.0.smithi079.stdout:1019: writing smithi07929762-14 from 1643418 to 2128262 tid 2 2024-04-24T01:59:19.803 INFO:tasks.rados.rados.0.smithi079.stdout:1020: snap_create 2024-04-24T01:59:19.819 INFO:tasks.rados.rados.0.smithi079.stdout:1019: finishing write tid 1 to smithi07929762-14 2024-04-24T01:59:19.833 INFO:tasks.rados.rados.0.smithi079.stdout:1019: finishing write tid 2 to smithi07929762-14 2024-04-24T01:59:19.836 INFO:tasks.rados.rados.0.smithi079.stdout:1019: finishing write tid 3 to smithi07929762-14 2024-04-24T01:59:19.837 INFO:tasks.rados.rados.0.smithi079.stdout:1019: finishing write tid 4 to smithi07929762-14 2024-04-24T01:59:19.837 INFO:tasks.rados.rados.0.smithi079.stdout:1019: oid 14 updating version 0 to 223 2024-04-24T01:59:19.837 INFO:tasks.rados.rados.0.smithi079.stdout:1019: oid 14 version 223 is already newer than 223 2024-04-24T01:59:19.837 INFO:tasks.rados.rados.0.smithi079.stdout:1019: oid 14 version 223 is already newer than 221 2024-04-24T01:59:19.837 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 14 v 223 (ObjNum 506 snap 112 seq_num 506) dirty exists 2024-04-24T01:59:19.837 INFO:tasks.rados.rados.0.smithi079.stdout:1019: left oid 14 (ObjNum 506 snap 112 seq_num 506) 2024-04-24T01:59:20.021 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T01:59:20.097 INFO:tasks.rados.rados.0.smithi079.stdout:1019: done (1 left) 2024-04-24T01:59:20.097 INFO:tasks.rados.rados.0.smithi079.stdout:1020: done (0 left) 2024-04-24T01:59:20.097 INFO:tasks.rados.rados.0.smithi079.stdout:1021: delete oid 1 current snap is 113 2024-04-24T01:59:20.142 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:59:20.139+0000 7fc0bab9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:59:20.149 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:59:20.148+0000 7fc0bab9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:59:20.158 INFO:tasks.rados.rados.0.smithi079.stdout:1021: done (0 left) 2024-04-24T01:59:20.158 INFO:tasks.rados.rados.0.smithi079.stdout:1022: write oid 25 current snap is 113 2024-04-24T01:59:20.158 INFO:tasks.rados.rados.0.smithi079.stdout:1022: seq_num 508 ranges {537956=705790,1644029=772160} 2024-04-24T01:59:20.165 INFO:tasks.rados.rados.0.smithi079.stdout:1022: writing smithi07929762-25 from 537956 to 1243746 tid 1 2024-04-24T01:59:20.172 INFO:tasks.rados.rados.0.smithi079.stdout:1022: writing smithi07929762-25 from 1644029 to 2416189 tid 2 2024-04-24T01:59:20.172 INFO:tasks.rados.rados.0.smithi079.stdout:1023: write oid 37 current snap is 113 2024-04-24T01:59:20.172 INFO:tasks.rados.rados.0.smithi079.stdout:1023: seq_num 509 ranges {554763=781039,2102511=452252} 2024-04-24T01:59:20.181 INFO:tasks.rados.rados.0.smithi079.stdout:1023: writing smithi07929762-37 from 554763 to 1335802 tid 1 2024-04-24T01:59:20.188 INFO:tasks.rados.rados.0.smithi079.stdout:1023: writing smithi07929762-37 from 2102511 to 2554763 tid 2 2024-04-24T01:59:20.188 INFO:tasks.rados.rados.0.smithi079.stdout:1024: snap_create 2024-04-24T01:59:20.257 INFO:tasks.rados.rados.0.smithi079.stdout:1022: finishing write tid 1 to smithi07929762-25 2024-04-24T01:59:20.258 INFO:tasks.rados.rados.0.smithi079.stdout:1023: finishing write tid 1 to smithi07929762-37 2024-04-24T01:59:20.260 INFO:tasks.rados.rados.0.smithi079.stdout:1022: finishing write tid 2 to smithi07929762-25 2024-04-24T01:59:20.262 INFO:tasks.rados.rados.0.smithi079.stdout:1022: finishing write tid 3 to smithi07929762-25 2024-04-24T01:59:20.263 INFO:tasks.rados.rados.0.smithi079.stdout:1022: finishing write tid 4 to smithi07929762-25 2024-04-24T01:59:20.263 INFO:tasks.rados.rados.0.smithi079.stdout:1022: oid 25 updating version 0 to 362 2024-04-24T01:59:20.263 INFO:tasks.rados.rados.0.smithi079.stdout:1022: oid 25 version 362 is already newer than 362 2024-04-24T01:59:20.263 INFO:tasks.rados.rados.0.smithi079.stdout:1022: oid 25 version 362 is already newer than 362 2024-04-24T01:59:20.263 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 25 v 362 (ObjNum 508 snap 113 seq_num 508) dirty exists 2024-04-24T01:59:20.263 INFO:tasks.rados.rados.0.smithi079.stdout:1022: left oid 25 (ObjNum 508 snap 113 seq_num 508) 2024-04-24T01:59:20.269 INFO:tasks.rados.rados.0.smithi079.stdout:1023: finishing write tid 2 to smithi07929762-37 2024-04-24T01:59:20.271 INFO:tasks.rados.rados.0.smithi079.stdout:1023: finishing write tid 3 to smithi07929762-37 2024-04-24T01:59:20.273 INFO:tasks.rados.rados.0.smithi079.stdout:1023: finishing write tid 4 to smithi07929762-37 2024-04-24T01:59:20.273 INFO:tasks.rados.rados.0.smithi079.stdout:1023: oid 37 updating version 0 to 465 2024-04-24T01:59:20.273 INFO:tasks.rados.rados.0.smithi079.stdout:1023: oid 37 version 465 is already newer than 465 2024-04-24T01:59:20.273 INFO:tasks.rados.rados.0.smithi079.stdout:1023: oid 37 version 465 is already newer than 465 2024-04-24T01:59:20.273 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 37 v 465 (ObjNum 509 snap 113 seq_num 509) dirty exists 2024-04-24T01:59:20.273 INFO:tasks.rados.rados.0.smithi079.stdout:1023: left oid 37 (ObjNum 509 snap 113 seq_num 509) 2024-04-24T01:59:21.047 INFO:tasks.rados.rados.0.smithi079.stdout:1022: done (2 left) 2024-04-24T01:59:21.047 INFO:tasks.rados.rados.0.smithi079.stdout:1023: done (1 left) 2024-04-24T01:59:21.047 INFO:tasks.rados.rados.0.smithi079.stdout:1024: done (0 left) 2024-04-24T01:59:21.047 INFO:tasks.rados.rados.0.smithi079.stdout:1025: write oid 23 current snap is 114 2024-04-24T01:59:21.047 INFO:tasks.rados.rados.0.smithi079.stdout:1025: seq_num 510 ranges {571570=572641,1877346=694224} 2024-04-24T01:59:21.052 INFO:tasks.rados.rados.0.smithi079.stdout:1025: writing smithi07929762-23 from 571570 to 1144211 tid 1 2024-04-24T01:59:21.061 INFO:tasks.rados.rados.0.smithi079.stdout:1025: writing smithi07929762-23 from 1877346 to 2571570 tid 2 2024-04-24T01:59:21.061 INFO:tasks.rados.rados.0.smithi079.stdout:1026: snap_create 2024-04-24T01:59:21.118 INFO:tasks.rados.rados.0.smithi079.stdout:1025: finishing write tid 1 to smithi07929762-23 2024-04-24T01:59:21.125 INFO:tasks.rados.rados.0.smithi079.stdout:1025: finishing write tid 2 to smithi07929762-23 2024-04-24T01:59:21.129 INFO:tasks.rados.rados.0.smithi079.stdout:1025: finishing write tid 3 to smithi07929762-23 2024-04-24T01:59:21.130 INFO:tasks.rados.rados.0.smithi079.stdout:1025: finishing write tid 4 to smithi07929762-23 2024-04-24T01:59:21.130 INFO:tasks.rados.rados.0.smithi079.stdout:1025: oid 23 updating version 0 to 81 2024-04-24T01:59:21.131 INFO:tasks.rados.rados.0.smithi079.stdout:1025: oid 23 version 81 is already newer than 81 2024-04-24T01:59:21.131 INFO:tasks.rados.rados.0.smithi079.stdout:1025: oid 23 version 81 is already newer than 81 2024-04-24T01:59:21.131 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 23 v 81 (ObjNum 510 snap 114 seq_num 510) dirty exists 2024-04-24T01:59:21.131 INFO:tasks.rados.rados.0.smithi079.stdout:1025: left oid 23 (ObjNum 510 snap 114 seq_num 510) 2024-04-24T01:59:22.049 INFO:tasks.rados.rados.0.smithi079.stdout:1025: done (1 left) 2024-04-24T01:59:22.049 INFO:tasks.rados.rados.0.smithi079.stdout:1026: done (0 left) 2024-04-24T01:59:22.049 INFO:tasks.rados.rados.0.smithi079.stdout:1027: read oid 17 snap -1 2024-04-24T01:59:22.049 INFO:tasks.rados.rados.0.smithi079.stdout:1027: expect deleted 2024-04-24T01:59:22.049 INFO:tasks.rados.rados.0.smithi079.stdout:1028: read oid 50 snap -1 2024-04-24T01:59:22.049 INFO:tasks.rados.rados.0.smithi079.stdout:1028: expect deleted 2024-04-24T01:59:22.049 INFO:tasks.rados.rados.0.smithi079.stdout:1029: write oid 48 current snap is 115 2024-04-24T01:59:22.049 INFO:tasks.rados.rados.0.smithi079.stdout:1029: seq_num 511 ranges {588377=647890,1935828=635840} 2024-04-24T01:59:22.055 INFO:tasks.rados.rados.0.smithi079.stdout:1029: writing smithi07929762-48 from 588377 to 1236267 tid 1 2024-04-24T01:59:22.064 INFO:tasks.rados.rados.0.smithi079.stdout:1029: writing smithi07929762-48 from 1935828 to 2571668 tid 2 2024-04-24T01:59:22.064 INFO:tasks.rados.rados.0.smithi079.stdout:1030: write oid 30 current snap is 115 2024-04-24T01:59:22.064 INFO:tasks.rados.rados.0.smithi079.stdout:1030: seq_num 512 ranges {605184=723139,1877957=727227} 2024-04-24T01:59:22.071 INFO:tasks.rados.rados.0.smithi079.stdout:1030: writing smithi07929762-30 from 605184 to 1328323 tid 1 2024-04-24T01:59:22.079 INFO:tasks.rados.rados.0.smithi079.stdout:1030: writing smithi07929762-30 from 1877957 to 2605184 tid 2 2024-04-24T01:59:22.079 INFO:tasks.rados.rados.0.smithi079.stdout:1031: snap_remove snap 111 2024-04-24T01:59:22.304 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T01:59:22.439 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:59:22.436+0000 7f22d7f5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:59:22.443 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:59:22.442+0000 7f22d7f5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:59:23.058 INFO:tasks.rados.rados.0.smithi079.stdout:1031: done (4 left) 2024-04-24T01:59:23.059 INFO:tasks.rados.rados.0.smithi079.stdout:1032: read oid 2 snap -1 2024-04-24T01:59:23.059 INFO:tasks.rados.rados.0.smithi079.stdout:1032: expect (ObjNum 442 snap 99 seq_num 442) 2024-04-24T01:59:23.059 INFO:tasks.rados.rados.0.smithi079.stdout:1033: read oid 29 snap -1 2024-04-24T01:59:23.059 INFO:tasks.rados.rados.0.smithi079.stdout:1033: expect deleted 2024-04-24T01:59:23.059 INFO:tasks.rados.rados.0.smithi079.stdout:1034: read oid 32 snap 114 2024-04-24T01:59:23.059 INFO:tasks.rados.rados.0.smithi079.stdout:1034: expect (ObjNum 374 snap 90 seq_num 374) 2024-04-24T01:59:23.059 INFO:tasks.rados.rados.0.smithi079.stdout:1035: delete oid 31 current snap is 115 2024-04-24T01:59:23.059 INFO:tasks.rados.rados.0.smithi079.stdout:1030: finishing write tid 1 to smithi07929762-30 2024-04-24T01:59:23.059 INFO:tasks.rados.rados.0.smithi079.stdout:1030: finishing write tid 2 to smithi07929762-30 2024-04-24T01:59:23.059 INFO:tasks.rados.rados.0.smithi079.stdout:1029: finishing write tid 1 to smithi07929762-48 2024-04-24T01:59:23.059 INFO:tasks.rados.rados.0.smithi079.stdout:1030: finishing write tid 3 to smithi07929762-30 2024-04-24T01:59:23.059 INFO:tasks.rados.rados.0.smithi079.stdout:1030: finishing write tid 4 to smithi07929762-30 2024-04-24T01:59:23.059 INFO:tasks.rados.rados.0.smithi079.stdout:1030: oid 30 updating version 0 to 106 2024-04-24T01:59:23.059 INFO:tasks.rados.rados.0.smithi079.stdout:1030: oid 30 version 106 is already newer than 106 2024-04-24T01:59:23.059 INFO:tasks.rados.rados.0.smithi079.stdout:1030: oid 30 version 106 is already newer than 106 2024-04-24T01:59:23.060 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 30 v 106 (ObjNum 512 snap 115 seq_num 512) dirty exists 2024-04-24T01:59:23.060 INFO:tasks.rados.rados.0.smithi079.stdout:1030: left oid 30 (ObjNum 512 snap 115 seq_num 512) 2024-04-24T01:59:23.060 INFO:tasks.rados.rados.0.smithi079.stdout:1029: finishing write tid 2 to smithi07929762-48 2024-04-24T01:59:23.060 INFO:tasks.rados.rados.0.smithi079.stdout:1029: finishing write tid 3 to smithi07929762-48 2024-04-24T01:59:23.060 INFO:tasks.rados.rados.0.smithi079.stdout:1029: finishing write tid 4 to smithi07929762-48 2024-04-24T01:59:23.060 INFO:tasks.rados.rados.0.smithi079.stdout:1029: oid 48 updating version 0 to 177 2024-04-24T01:59:23.060 INFO:tasks.rados.rados.0.smithi079.stdout:1029: oid 48 version 177 is already newer than 177 2024-04-24T01:59:23.060 INFO:tasks.rados.rados.0.smithi079.stdout:1029: oid 48 version 177 is already newer than 177 2024-04-24T01:59:23.060 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 48 v 177 (ObjNum 511 snap 115 seq_num 511) dirty exists 2024-04-24T01:59:23.060 INFO:tasks.rados.rados.0.smithi079.stdout:1029: left oid 48 (ObjNum 511 snap 115 seq_num 511) 2024-04-24T01:59:23.147 INFO:tasks.rados.rados.0.smithi079.stdout:1032: expect (ObjNum 442 snap 99 seq_num 442) 2024-04-24T01:59:23.366 INFO:tasks.rados.rados.0.smithi079.stdout:1034: expect (ObjNum 374 snap 90 seq_num 374) 2024-04-24T01:59:23.482 INFO:tasks.rados.rados.0.smithi079.stdout:1027: done (7 left) 2024-04-24T01:59:23.482 INFO:tasks.rados.rados.0.smithi079.stdout:1028: done (6 left) 2024-04-24T01:59:23.482 INFO:tasks.rados.rados.0.smithi079.stdout:1029: done (5 left) 2024-04-24T01:59:23.482 INFO:tasks.rados.rados.0.smithi079.stdout:1030: done (4 left) 2024-04-24T01:59:23.482 INFO:tasks.rados.rados.0.smithi079.stdout:1032: done (3 left) 2024-04-24T01:59:23.482 INFO:tasks.rados.rados.0.smithi079.stdout:1033: done (2 left) 2024-04-24T01:59:23.482 INFO:tasks.rados.rados.0.smithi079.stdout:1034: done (1 left) 2024-04-24T01:59:23.482 INFO:tasks.rados.rados.0.smithi079.stdout:1035: done (0 left) 2024-04-24T01:59:23.482 INFO:tasks.rados.rados.0.smithi079.stdout:1036: write oid 16 current snap is 115 2024-04-24T01:59:23.483 INFO:tasks.rados.rados.0.smithi079.stdout:1036: seq_num 514 ranges {638798=473637,1594921=783167} 2024-04-24T01:59:23.487 INFO:tasks.rados.rados.0.smithi079.stdout:1036: writing smithi07929762-16 from 638798 to 1112435 tid 1 2024-04-24T01:59:23.497 INFO:tasks.rados.rados.0.smithi079.stdout:1036: writing smithi07929762-16 from 1594921 to 2378088 tid 2 2024-04-24T01:59:23.497 INFO:tasks.rados.rados.0.smithi079.stdout:1037: read oid 7 snap -1 2024-04-24T01:59:23.497 INFO:tasks.rados.rados.0.smithi079.stdout:1037: expect (ObjNum 375 snap 90 seq_num 375) 2024-04-24T01:59:23.497 INFO:tasks.rados.rados.0.smithi079.stdout:1038: snap_remove snap 112 2024-04-24T01:59:24.057 INFO:tasks.rados.rados.0.smithi079.stdout:1038: done (2 left) 2024-04-24T01:59:24.057 INFO:tasks.rados.rados.0.smithi079.stdout:1039: write oid 17 current snap is 115 2024-04-24T01:59:24.057 INFO:tasks.rados.rados.0.smithi079.stdout:1039: seq_num 515 ranges {655605=548886,1653403=643178} 2024-04-24T01:59:24.063 INFO:tasks.rados.rados.0.smithi079.stdout:1039: writing smithi07929762-17 from 655605 to 1204491 tid 1 2024-04-24T01:59:24.069 INFO:tasks.rados.rados.0.smithi079.stdout:1039: writing smithi07929762-17 from 1653403 to 2296581 tid 2 2024-04-24T01:59:24.069 INFO:tasks.rados.rados.0.smithi079.stdout:1040: delete oid 42 current snap is 115 2024-04-24T01:59:24.069 INFO:tasks.rados.rados.0.smithi079.stdout:1036: finishing write tid 1 to smithi07929762-16 2024-04-24T01:59:24.069 INFO:tasks.rados.rados.0.smithi079.stdout:1037: expect (ObjNum 375 snap 90 seq_num 375) 2024-04-24T01:59:24.178 INFO:tasks.rados.rados.0.smithi079.stdout:1036: finishing write tid 2 to smithi07929762-16 2024-04-24T01:59:24.178 INFO:tasks.rados.rados.0.smithi079.stdout:1036: finishing write tid 3 to smithi07929762-16 2024-04-24T01:59:24.178 INFO:tasks.rados.rados.0.smithi079.stdout:1036: finishing write tid 4 to smithi07929762-16 2024-04-24T01:59:24.178 INFO:tasks.rados.rados.0.smithi079.stdout:1036: oid 16 updating version 0 to 475 2024-04-24T01:59:24.178 INFO:tasks.rados.rados.0.smithi079.stdout:1036: oid 16 version 475 is already newer than 475 2024-04-24T01:59:24.178 INFO:tasks.rados.rados.0.smithi079.stdout:1036: oid 16 version 475 is already newer than 475 2024-04-24T01:59:24.179 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 16 v 475 (ObjNum 514 snap 115 seq_num 514) dirty exists 2024-04-24T01:59:24.179 INFO:tasks.rados.rados.0.smithi079.stdout:1036: left oid 16 (ObjNum 514 snap 115 seq_num 514) 2024-04-24T01:59:24.179 INFO:tasks.rados.rados.0.smithi079.stdout:1039: finishing write tid 1 to smithi07929762-17 2024-04-24T01:59:24.179 INFO:tasks.rados.rados.0.smithi079.stdout:1039: finishing write tid 2 to smithi07929762-17 2024-04-24T01:59:24.185 INFO:tasks.rados.rados.0.smithi079.stdout:1039: finishing write tid 3 to smithi07929762-17 2024-04-24T01:59:24.186 INFO:tasks.rados.rados.0.smithi079.stdout:1039: finishing write tid 4 to smithi07929762-17 2024-04-24T01:59:24.186 INFO:tasks.rados.rados.0.smithi079.stdout:1039: oid 17 updating version 0 to 240 2024-04-24T01:59:24.186 INFO:tasks.rados.rados.0.smithi079.stdout:1039: oid 17 version 240 is already newer than 240 2024-04-24T01:59:24.187 INFO:tasks.rados.rados.0.smithi079.stdout:1039: oid 17 version 240 is already newer than 240 2024-04-24T01:59:24.187 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 17 v 240 (ObjNum 515 snap 115 seq_num 515) dirty exists 2024-04-24T01:59:24.187 INFO:tasks.rados.rados.0.smithi079.stdout:1039: left oid 17 (ObjNum 515 snap 115 seq_num 515) 2024-04-24T01:59:24.223 INFO:tasks.rados.rados.0.smithi079.stdout:1036: done (3 left) 2024-04-24T01:59:24.224 INFO:tasks.rados.rados.0.smithi079.stdout:1037: done (2 left) 2024-04-24T01:59:24.224 INFO:tasks.rados.rados.0.smithi079.stdout:1039: done (1 left) 2024-04-24T01:59:24.224 INFO:tasks.rados.rados.0.smithi079.stdout:1040: done (0 left) 2024-04-24T01:59:24.224 INFO:tasks.rados.rados.0.smithi079.stdout:1041: write oid 47 current snap is 115 2024-04-24T01:59:24.224 INFO:tasks.rados.rados.0.smithi079.stdout:1041: seq_num 517 ranges {689219=415737,1770367=646847} 2024-04-24T01:59:24.230 INFO:tasks.rados.rados.0.smithi079.stdout:1041: writing smithi07929762-47 from 689219 to 1104956 tid 1 2024-04-24T01:59:24.239 INFO:tasks.rados.rados.0.smithi079.stdout:1041: writing smithi07929762-47 from 1770367 to 2417214 tid 2 2024-04-24T01:59:24.239 INFO:tasks.rados.rados.0.smithi079.stdout:1042: write (excl) oid 35 current snap is 115 2024-04-24T01:59:24.239 INFO:tasks.rados.rados.0.smithi079.stdout:1042: seq_num 518 ranges {706026=490986,1828849=790505} 2024-04-24T01:59:24.247 INFO:tasks.rados.rados.0.smithi079.stdout:1042: writing smithi07929762-35 from 706026 to 1197012 tid 1 2024-04-24T01:59:24.260 INFO:tasks.rados.rados.0.smithi079.stdout:1042: writing smithi07929762-35 from 1828849 to 2619354 tid 2 2024-04-24T01:59:24.260 INFO:tasks.rados.rados.0.smithi079.stdout:1043: read oid 2 snap 113 2024-04-24T01:59:24.260 INFO:tasks.rados.rados.0.smithi079.stdout:1043: expect (ObjNum 442 snap 99 seq_num 442) 2024-04-24T01:59:24.260 INFO:tasks.rados.rados.0.smithi079.stdout:1044: delete oid 8 current snap is 115 2024-04-24T01:59:24.290 INFO:tasks.rados.rados.0.smithi079.stdout:1043: expect (ObjNum 442 snap 99 seq_num 442) 2024-04-24T01:59:24.509 INFO:tasks.rados.rados.0.smithi079.stdout:1041: finishing write tid 1 to smithi07929762-47 2024-04-24T01:59:24.509 INFO:tasks.rados.rados.0.smithi079.stdout:1041: finishing write tid 2 to smithi07929762-47 2024-04-24T01:59:24.509 INFO:tasks.rados.rados.0.smithi079.stdout:1042: finishing write tid 1 to smithi07929762-35 2024-04-24T01:59:24.509 INFO:tasks.rados.rados.0.smithi079.stdout:1041: finishing write tid 3 to smithi07929762-47 2024-04-24T01:59:24.509 INFO:tasks.rados.rados.0.smithi079.stdout:1041: finishing write tid 4 to smithi07929762-47 2024-04-24T01:59:24.509 INFO:tasks.rados.rados.0.smithi079.stdout:1041: oid 47 updating version 0 to 369 2024-04-24T01:59:24.509 INFO:tasks.rados.rados.0.smithi079.stdout:1041: oid 47 version 369 is already newer than 369 2024-04-24T01:59:24.509 INFO:tasks.rados.rados.0.smithi079.stdout:1041: oid 47 version 369 is already newer than 369 2024-04-24T01:59:24.509 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 47 v 369 (ObjNum 517 snap 115 seq_num 517) dirty exists 2024-04-24T01:59:24.509 INFO:tasks.rados.rados.0.smithi079.stdout:1041: left oid 47 (ObjNum 517 snap 115 seq_num 517) 2024-04-24T01:59:24.509 INFO:tasks.rados.rados.0.smithi079.stdout:1042: finishing write tid 2 to smithi07929762-35 2024-04-24T01:59:24.509 INFO:tasks.rados.rados.0.smithi079.stdout:1041: done (3 left) 2024-04-24T01:59:24.509 INFO:tasks.rados.rados.0.smithi079.stdout:1043: done (2 left) 2024-04-24T01:59:24.510 INFO:tasks.rados.rados.0.smithi079.stdout:1044: done (1 left) 2024-04-24T01:59:24.510 INFO:tasks.rados.rados.0.smithi079.stdout:1045: snap_create 2024-04-24T01:59:24.510 INFO:tasks.rados.rados.0.smithi079.stdout:1042: finishing write tid 3 to smithi07929762-35 2024-04-24T01:59:24.510 INFO:tasks.rados.rados.0.smithi079.stdout:1042: finishing write tid 4 to smithi07929762-35 2024-04-24T01:59:24.510 INFO:tasks.rados.rados.0.smithi079.stdout:1042: oid 35 updating version 0 to 303 2024-04-24T01:59:24.510 INFO:tasks.rados.rados.0.smithi079.stdout:1042: oid 35 updating version 303 to 305 2024-04-24T01:59:24.510 INFO:tasks.rados.rados.0.smithi079.stdout:1042: oid 35 version 305 is already newer than 305 2024-04-24T01:59:24.510 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 35 v 305 (ObjNum 518 snap 115 seq_num 518) dirty exists 2024-04-24T01:59:24.510 INFO:tasks.rados.rados.0.smithi079.stdout:1042: left oid 35 (ObjNum 518 snap 115 seq_num 518) 2024-04-24T01:59:24.535 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T01:59:24.607 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-04-24T01:59:24.607 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-04-24T01:59:24.607 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T01:59:24.640 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:59:24.638+0000 7f384026d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:59:24.644 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:59:24.643+0000 7f384026d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:59:24.721 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:59:24.717+0000 7f374859d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:59:24.725 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:59:24.724+0000 7f374859d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:59:24.916 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:59:24.917 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-24T01:59:24.930 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":363,"stamp":"2024-04-24T01:59:24.055819+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":80896359,"num_objects":36,"num_object_clones":5,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":1916,"num_read_kb":1272344,"num_write":1784,"num_write_kb":450791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3431,"ondisk_log_size":3431,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":326892,"kb_used_data":175804,"kb_used_omap":0,"kb_used_meta":140544,"kb_avail":377160468,"statfs":{"total":386547056640,"available":386212319232,"internally_reserved":0,"allocated":180023296,"data_stored":181262612,"data_compressed":89889,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":143917056},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-1495757,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":6989,"num_write":9,"num_write_kb":2462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.960263"},"pg_stats":[{"pgid":"3.f","version":"233'173","reported_seq":894,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-24T01:59:18.110392+0000","last_change":"2024-04-24T01:59:17.081594+0000","last_active":"2024-04-24T01:59:18.110392+0000","last_peered":"2024-04-24T01:59:18.110392+0000","last_clean":"2024-04-24T01:59:18.110392+0000","last_became_active":"2024-04-24T01:55:30.386726+0000","last_became_peered":"2024-04-24T01:55:30.386726+0000","last_unstale":"2024-04-24T01:59:18.110392+0000","last_undegraded":"2024-04-24T01:59:18.110392+0000","last_fullsized":"2024-04-24T01:59:18.110392+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":173,"log_dups_size":0,"ondisk_log_size":173,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":17,"snaptrim_duration":0,"stat_sum":{"num_bytes":6151494,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":115,"num_read_kb":98571,"num_write":84,"num_write_kb":23977,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"242'290","reported_seq":972,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-24T01:59:18.318326+0000","last_change":"2024-04-24T01:59:18.109797+0000","last_active":"2024-04-24T01:59:18.318326+0000","last_peered":"2024-04-24T01:59:18.318326+0000","last_clean":"2024-04-24T01:59:18.318326+0000","last_became_active":"2024-04-24T01:55:30.321866+0000","last_became_peered":"2024-04-24T01:55:30.321866+0000","last_unstale":"2024-04-24T01:59:18.318326+0000","last_undegraded":"2024-04-24T01:59:18.318326+0000","last_fullsized":"2024-04-24T01:59:18.318326+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":290,"log_dups_size":0,"ondisk_log_size":290,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":26,"snaptrim_duration":0,"stat_sum":{"num_bytes":5966617,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":162,"num_read_kb":113139,"num_write":142,"num_write_kb":38189,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"242'223","reported_seq":929,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-04-24T01:59:23.073732+0000","last_change":"2024-04-24T01:59:23.081237+0000","last_active":"2024-04-24T01:59:23.073732+0000","last_peered":"2024-04-24T01:59:23.073732+0000","last_clean":"2024-04-24T01:59:23.073732+0000","last_became_active":"2024-04-24T01:55:30.320186+0000","last_became_peered":"2024-04-24T01:55:30.320186+0000","last_unstale":"2024-04-24T01:59:23.073732+0000","last_undegraded":"2024-04-24T01:59:23.073732+0000","last_fullsized":"2024-04-24T01:59:23.073732+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":223,"log_dups_size":0,"ondisk_log_size":223,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":22,"snaptrim_duration":0,"stat_sum":{"num_bytes":8454152,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":118,"num_read_kb":79388,"num_write":123,"num_write_kb":34018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"242'257","reported_seq":959,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-24T01:59:18.329851+0000","last_change":"2024-04-24T01:59:18.113663+0000","last_active":"2024-04-24T01:59:18.329851+0000","last_peered":"2024-04-24T01:59:18.329851+0000","last_clean":"2024-04-24T01:59:18.329851+0000","last_became_active":"2024-04-24T01:55:30.288081+0000","last_became_peered":"2024-04-24T01:55:30.288081+0000","last_unstale":"2024-04-24T01:59:18.329851+0000","last_undegraded":"2024-04-24T01:59:18.329851+0000","last_fullsized":"2024-04-24T01:59:18.329851+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":257,"log_dups_size":0,"ondisk_log_size":257,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":26,"snaptrim_duration":0,"stat_sum":{"num_bytes":2269044,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":134,"num_read_kb":92552,"num_write":124,"num_write_kb":33091,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"242'83","reported_seq":791,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-24T01:59:18.173226+0000","last_change":"2024-04-24T01:59:18.298093+0000","last_active":"2024-04-24T01:59:18.173226+0000","last_peered":"2024-04-24T01:59:18.173226+0000","last_clean":"2024-04-24T01:59:18.173226+0000","last_became_active":"2024-04-24T01:55:30.333815+0000","last_became_peered":"2024-04-24T01:55:30.333815+0000","last_unstale":"2024-04-24T01:59:18.173226+0000","last_undegraded":"2024-04-24T01:59:18.173226+0000","last_fullsized":"2024-04-24T01:59:18.173226+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":83,"log_dups_size":0,"ondisk_log_size":83,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":7,"snaptrim_duration":0,"stat_sum":{"num_bytes":2050553,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":42,"num_read_kb":23145,"num_write":51,"num_write_kb":14288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"241'451","reported_seq":1120,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-24T01:59:18.248989+0000","last_change":"2024-04-24T01:59:17.079049+0000","last_active":"2024-04-24T01:59:18.248989+0000","last_peered":"2024-04-24T01:59:18.248989+0000","last_clean":"2024-04-24T01:59:18.248989+0000","last_became_active":"2024-04-24T01:55:30.315075+0000","last_became_peered":"2024-04-24T01:55:30.315075+0000","last_unstale":"2024-04-24T01:59:18.248989+0000","last_undegraded":"2024-04-24T01:59:18.248989+0000","last_fullsized":"2024-04-24T01:59:18.248989+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":451,"log_dups_size":0,"ondisk_log_size":451,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":42,"snaptrim_duration":0,"stat_sum":{"num_bytes":10185372,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":218,"num_read_kb":141513,"num_write":199,"num_write_kb":52653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"233'78","reported_seq":800,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-24T01:59:18.174148+0000","last_change":"2024-04-24T01:59:17.081365+0000","last_active":"2024-04-24T01:59:18.174148+0000","last_peered":"2024-04-24T01:59:18.174148+0000","last_clean":"2024-04-24T01:59:18.174148+0000","last_became_active":"2024-04-24T01:55:30.350300+0000","last_became_peered":"2024-04-24T01:55:30.350300+0000","last_unstale":"2024-04-24T01:59:18.174148+0000","last_undegraded":"2024-04-24T01:59:18.174148+0000","last_fullsized":"2024-04-24T01:59:18.174148+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":8,"snaptrim_duration":0,"stat_sum":{"num_bytes":3916097,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":43,"num_read_kb":22972,"num_write":52,"num_write_kb":13195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"230'24","reported_seq":743,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-04-24T01:59:23.083598+0000","last_change":"2024-04-24T01:59:23.093633+0000","last_active":"2024-04-24T01:59:23.083598+0000","last_peered":"2024-04-24T01:59:23.083598+0000","last_clean":"2024-04-24T01:59:23.083598+0000","last_became_active":"2024-04-24T01:55:30.376813+0000","last_became_peered":"2024-04-24T01:55:30.376813+0000","last_unstale":"2024-04-24T01:59:23.083598+0000","last_undegraded":"2024-04-24T01:59:23.083598+0000","last_fullsized":"2024-04-24T01:59:23.083598+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0,"stat_sum":{"num_bytes":3798448,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":13,"num_read_kb":10081,"num_write":21,"num_write_kb":5614,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'97","reported_seq":742,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-24T01:59:18.128835+0000","last_change":"2024-04-24T01:55:16.196470+0000","last_active":"2024-04-24T01:59:18.128835+0000","last_peered":"2024-04-24T01:59:18.128835+0000","last_clean":"2024-04-24T01:59:18.128835+0000","last_became_active":"2024-04-24T01:55:16.190464+0000","last_became_peered":"2024-04-24T01:55:16.190464+0000","last_unstale":"2024-04-24T01:59:18.128835+0000","last_undegraded":"2024-04-24T01:59:18.128835+0000","last_fullsized":"2024-04-24T01:59:18.128835+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:15.127115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:15.127115+0000","last_clean_scrub_stamp":"2024-04-24T01:55:15.127115+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":615,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-04-24T01:59:23.087928+0000","last_change":"2024-04-24T01:55:18.236148+0000","last_active":"2024-04-24T01:59:23.087928+0000","last_peered":"2024-04-24T01:59:23.087928+0000","last_clean":"2024-04-24T01:59:23.087928+0000","last_became_active":"2024-04-24T01:55:18.230154+0000","last_became_peered":"2024-04-24T01:55:18.230154+0000","last_unstale":"2024-04-24T01:59:23.087928+0000","last_undegraded":"2024-04-24T01:59:23.087928+0000","last_fullsized":"2024-04-24T01:59:23.087928+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"246'364","reported_seq":1069,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-04-24T01:59:23.107441+0000","last_change":"2024-04-24T01:59:23.165114+0000","last_active":"2024-04-24T01:59:23.107441+0000","last_peered":"2024-04-24T01:59:23.107441+0000","last_clean":"2024-04-24T01:59:23.107441+0000","last_became_active":"2024-04-24T01:55:30.316222+0000","last_became_peered":"2024-04-24T01:55:30.316222+0000","last_unstale":"2024-04-24T01:59:23.107441+0000","last_undegraded":"2024-04-24T01:59:23.107441+0000","last_fullsized":"2024-04-24T01:59:23.107441+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":364,"log_dups_size":0,"ondisk_log_size":364,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":28,"snaptrim_duration":0,"stat_sum":{"num_bytes":6235562,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":196,"num_read_kb":136827,"num_write":191,"num_write_kb":53760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":619,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-24T01:59:18.111027+0000","last_change":"2024-04-24T01:55:18.265905+0000","last_active":"2024-04-24T01:59:18.111027+0000","last_peered":"2024-04-24T01:59:18.111027+0000","last_clean":"2024-04-24T01:59:18.111027+0000","last_became_active":"2024-04-24T01:55:18.257020+0000","last_became_peered":"2024-04-24T01:55:18.257020+0000","last_unstale":"2024-04-24T01:59:18.111027+0000","last_undegraded":"2024-04-24T01:59:18.111027+0000","last_fullsized":"2024-04-24T01:59:18.111027+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"239'298","reported_seq":970,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-24T01:59:18.132976+0000","last_change":"2024-04-24T01:59:18.144568+0000","last_active":"2024-04-24T01:59:18.132976+0000","last_peered":"2024-04-24T01:59:18.132976+0000","last_clean":"2024-04-24T01:59:18.132976+0000","last_became_active":"2024-04-24T01:55:30.344129+0000","last_became_peered":"2024-04-24T01:55:30.344129+0000","last_unstale":"2024-04-24T01:59:18.132976+0000","last_undegraded":"2024-04-24T01:59:18.132976+0000","last_fullsized":"2024-04-24T01:59:18.132976+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":298,"log_dups_size":0,"ondisk_log_size":298,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":24,"snaptrim_duration":0,"stat_sum":{"num_bytes":2033746,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":127,"num_read_kb":72536,"num_write":135,"num_write_kb":37841,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":615,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-04-24T01:59:23.076715+0000","last_change":"2024-04-24T01:55:18.215251+0000","last_active":"2024-04-24T01:59:23.076715+0000","last_peered":"2024-04-24T01:59:23.076715+0000","last_clean":"2024-04-24T01:59:23.076715+0000","last_became_active":"2024-04-24T01:55:18.206605+0000","last_became_peered":"2024-04-24T01:55:18.206605+0000","last_unstale":"2024-04-24T01:59:23.076715+0000","last_undegraded":"2024-04-24T01:59:23.076715+0000","last_fullsized":"2024-04-24T01:59:23.076715+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"242'142","reported_seq":861,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-24T01:59:18.322114+0000","last_change":"2024-04-24T01:59:18.153317+0000","last_active":"2024-04-24T01:59:18.322114+0000","last_peered":"2024-04-24T01:59:18.322114+0000","last_clean":"2024-04-24T01:59:18.322114+0000","last_became_active":"2024-04-24T01:55:30.401366+0000","last_became_peered":"2024-04-24T01:55:30.401366+0000","last_unstale":"2024-04-24T01:59:18.322114+0000","last_undegraded":"2024-04-24T01:59:18.322114+0000","last_fullsized":"2024-04-24T01:59:18.322114+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":10,"snaptrim_duration":0,"stat_sum":{"num_bytes":5563282,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":118,"num_read_kb":89048,"num_write":55,"num_write_kb":14247,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":619,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-24T01:59:18.120319+0000","last_change":"2024-04-24T01:55:18.230893+0000","last_active":"2024-04-24T01:59:18.120319+0000","last_peered":"2024-04-24T01:59:18.120319+0000","last_clean":"2024-04-24T01:59:18.120319+0000","last_became_active":"2024-04-24T01:55:18.225188+0000","last_became_peered":"2024-04-24T01:55:18.225188+0000","last_unstale":"2024-04-24T01:59:18.120319+0000","last_undegraded":"2024-04-24T01:59:18.120319+0000","last_fullsized":"2024-04-24T01:59:18.120319+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"241'238","reported_seq":907,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-24T01:59:18.268075+0000","last_change":"2024-04-24T01:59:18.104347+0000","last_active":"2024-04-24T01:59:18.268075+0000","last_peered":"2024-04-24T01:59:18.268075+0000","last_clean":"2024-04-24T01:59:18.268075+0000","last_became_active":"2024-04-24T01:55:30.343318+0000","last_became_peered":"2024-04-24T01:55:30.343318+0000","last_unstale":"2024-04-24T01:59:18.268075+0000","last_undegraded":"2024-04-24T01:59:18.268075+0000","last_fullsized":"2024-04-24T01:59:18.268075+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":238,"log_dups_size":0,"ondisk_log_size":238,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":21,"snaptrim_duration":0,"stat_sum":{"num_bytes":6252336,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":114,"num_read_kb":92781,"num_write":107,"num_write_kb":30307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"246'316","reported_seq":1012,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-04-24T01:59:23.097005+0000","last_change":"2024-04-24T01:59:23.165819+0000","last_active":"2024-04-24T01:59:23.097005+0000","last_peered":"2024-04-24T01:59:23.097005+0000","last_clean":"2024-04-24T01:59:23.097005+0000","last_became_active":"2024-04-24T01:55:30.345308+0000","last_became_peered":"2024-04-24T01:55:30.345308+0000","last_unstale":"2024-04-24T01:59:23.097005+0000","last_undegraded":"2024-04-24T01:59:23.097005+0000","last_fullsized":"2024-04-24T01:59:23.097005+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":316,"log_dups_size":0,"ondisk_log_size":316,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":27,"snaptrim_duration":0,"stat_sum":{"num_bytes":4386825,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":167,"num_read_kb":110065,"num_write":148,"num_write_kb":39126,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":593,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-24T01:59:18.137949+0000","last_change":"2024-04-24T01:55:18.224238+0000","last_active":"2024-04-24T01:59:18.137949+0000","last_peered":"2024-04-24T01:59:18.137949+0000","last_clean":"2024-04-24T01:59:18.137949+0000","last_became_active":"2024-04-24T01:55:18.219131+0000","last_became_peered":"2024-04-24T01:55:18.219131+0000","last_unstale":"2024-04-24T01:59:18.137949+0000","last_undegraded":"2024-04-24T01:59:18.137949+0000","last_fullsized":"2024-04-24T01:59:18.137949+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"217'58","reported_seq":753,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-24T01:59:18.129526+0000","last_change":"2024-04-24T01:59:17.083176+0000","last_active":"2024-04-24T01:59:18.129526+0000","last_peered":"2024-04-24T01:59:18.129526+0000","last_clean":"2024-04-24T01:59:18.129526+0000","last_became_active":"2024-04-24T01:55:30.319958+0000","last_became_peered":"2024-04-24T01:55:30.319958+0000","last_unstale":"2024-04-24T01:59:18.129526+0000","last_undegraded":"2024-04-24T01:59:18.129526+0000","last_fullsized":"2024-04-24T01:59:18.129526+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":5,"snaptrim_duration":0,"stat_sum":{"num_bytes":3428694,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":27,"num_read_kb":19609,"num_write":42,"num_write_kb":11889,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":609,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-24T01:59:18.147505+0000","last_change":"2024-04-24T01:55:18.235763+0000","last_active":"2024-04-24T01:59:18.147505+0000","last_peered":"2024-04-24T01:59:18.147505+0000","last_clean":"2024-04-24T01:59:18.147505+0000","last_became_active":"2024-04-24T01:55:18.229175+0000","last_became_peered":"2024-04-24T01:55:18.229175+0000","last_unstale":"2024-04-24T01:59:18.147505+0000","last_undegraded":"2024-04-24T01:59:18.147505+0000","last_fullsized":"2024-04-24T01:59:18.147505+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"223'102","reported_seq":803,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-24T01:59:18.249814+0000","last_change":"2024-04-24T01:59:18.143970+0000","last_active":"2024-04-24T01:59:18.249814+0000","last_peered":"2024-04-24T01:59:18.249814+0000","last_clean":"2024-04-24T01:59:18.249814+0000","last_became_active":"2024-04-24T01:55:30.372690+0000","last_became_peered":"2024-04-24T01:55:30.372690+0000","last_unstale":"2024-04-24T01:59:18.249814+0000","last_undegraded":"2024-04-24T01:59:18.249814+0000","last_fullsized":"2024-04-24T01:59:18.249814+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":9,"snaptrim_duration":0,"stat_sum":{"num_bytes":3630378,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":54,"num_read_kb":39371,"num_write":45,"num_write_kb":12824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":593,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-24T01:59:18.130304+0000","last_change":"2024-04-24T01:55:18.278351+0000","last_active":"2024-04-24T01:59:18.130304+0000","last_peered":"2024-04-24T01:59:18.130304+0000","last_clean":"2024-04-24T01:59:18.130304+0000","last_became_active":"2024-04-24T01:55:18.273668+0000","last_became_peered":"2024-04-24T01:55:18.273668+0000","last_unstale":"2024-04-24T01:59:18.130304+0000","last_undegraded":"2024-04-24T01:59:18.130304+0000","last_fullsized":"2024-04-24T01:59:18.130304+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"235'237","reported_seq":939,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-24T01:59:18.227196+0000","last_change":"2024-04-24T01:59:17.083895+0000","last_active":"2024-04-24T01:59:18.227196+0000","last_peered":"2024-04-24T01:59:18.227196+0000","last_clean":"2024-04-24T01:59:18.227196+0000","last_became_active":"2024-04-24T01:55:30.363123+0000","last_became_peered":"2024-04-24T01:55:30.363123+0000","last_unstale":"2024-04-24T01:59:18.227196+0000","last_undegraded":"2024-04-24T01:59:18.227196+0000","last_fullsized":"2024-04-24T01:59:18.227196+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":237,"log_dups_size":0,"ondisk_log_size":237,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":27,"snaptrim_duration":0,"stat_sum":{"num_bytes":5983391,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":165,"num_read_kb":130655,"num_write":128,"num_write_kb":33878,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":615,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-04-24T01:59:23.088603+0000","last_change":"2024-04-24T01:55:18.241117+0000","last_active":"2024-04-24T01:59:23.088603+0000","last_peered":"2024-04-24T01:59:23.088603+0000","last_clean":"2024-04-24T01:59:23.088603+0000","last_became_active":"2024-04-24T01:55:18.234313+0000","last_became_peered":"2024-04-24T01:55:18.234313+0000","last_unstale":"2024-04-24T01:59:23.088603+0000","last_undegraded":"2024-04-24T01:59:23.088603+0000","last_fullsized":"2024-04-24T01:59:23.088603+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":80305991,"num_objects":34,"num_object_clones":5,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":1813,"num_read_kb":1272253,"num_write":1647,"num_write_kb":448897,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":166944768,"data_stored":170776149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3334,"ondisk_log_size":3334,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":1}],"osd_stats":[{"osd":3,"up_from":10,"seq":42949673204,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":87208,"kb_used_data":47496,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94284632,"statfs":{"total":96636764160,"available":96547463168,"internally_reserved":0,"allocated":48635904,"data_stored":49109180,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673204,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":80500,"kb_used_data":40788,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94291340,"statfs":{"total":96636764160,"available":96554332160,"internally_reserved":0,"allocated":41766912,"data_stored":42319490,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":70832,"kb_used_data":34312,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94301008,"statfs":{"total":96636764160,"available":96564232192,"internally_reserved":0,"allocated":35135488,"data_stored":35016658,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705913,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":88352,"kb_used_data":53208,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94283488,"statfs":{"total":96636764160,"available":96546291712,"internally_reserved":0,"allocated":54484992,"data_stored":54817284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":33087488,"data_stored":33562100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":47493120,"data_stored":48660848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":39776256,"data_stored":40898579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":46587904,"data_stored":47654622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T01:59:25.065 INFO:tasks.rados.rados.0.smithi079.stdout:1042: done (1 left) 2024-04-24T01:59:25.065 INFO:tasks.rados.rados.0.smithi079.stdout:1045: done (0 left) 2024-04-24T01:59:25.065 INFO:tasks.rados.rados.0.smithi079.stdout:1046: read oid 48 snap -1 2024-04-24T01:59:25.065 INFO:tasks.rados.rados.0.smithi079.stdout:1046: expect (ObjNum 511 snap 115 seq_num 511) 2024-04-24T01:59:25.065 INFO:tasks.rados.rados.0.smithi079.stdout:1047: write (excl) oid 45 current snap is 116 2024-04-24T01:59:25.065 INFO:tasks.rados.rados.0.smithi079.stdout:1047: seq_num 520 ranges {739640=641484,1829460=794174} 2024-04-24T01:59:25.072 INFO:tasks.rados.rados.0.smithi079.stdout:1047: writing smithi07929762-45 from 739640 to 1381124 tid 1 2024-04-24T01:59:25.081 INFO:tasks.rados.rados.0.smithi079.stdout:1047: writing smithi07929762-45 from 1829460 to 2623634 tid 2 2024-04-24T01:59:25.081 INFO:tasks.rados.rados.0.smithi079.stdout:1048: delete oid 9 current snap is 116 2024-04-24T01:59:25.137 INFO:tasks.rados.rados.0.smithi079.stdout:1046: expect (ObjNum 511 snap 115 seq_num 511) 2024-04-24T01:59:25.309 INFO:tasks.rados.rados.0.smithi079.stdout:1047: finishing write tid 1 to smithi07929762-45 2024-04-24T01:59:25.309 INFO:tasks.rados.rados.0.smithi079.stdout:1047: finishing write tid 2 to smithi07929762-45 2024-04-24T01:59:25.309 INFO:tasks.rados.rados.0.smithi079.stdout:1047: finishing write tid 3 to smithi07929762-45 2024-04-24T01:59:25.309 INFO:tasks.rados.rados.0.smithi079.stdout:1047: finishing write tid 4 to smithi07929762-45 2024-04-24T01:59:25.309 INFO:tasks.rados.rados.0.smithi079.stdout:1047: oid 45 updating version 0 to 299 2024-04-24T01:59:25.309 INFO:tasks.rados.rados.0.smithi079.stdout:1047: oid 45 version 299 is already newer than 299 2024-04-24T01:59:25.310 INFO:tasks.rados.rados.0.smithi079.stdout:1047: oid 45 version 299 is already newer than 297 2024-04-24T01:59:25.310 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 45 v 299 (ObjNum 520 snap 116 seq_num 520) dirty exists 2024-04-24T01:59:25.310 INFO:tasks.rados.rados.0.smithi079.stdout:1047: left oid 45 (ObjNum 520 snap 116 seq_num 520) 2024-04-24T01:59:25.310 INFO:tasks.rados.rados.0.smithi079.stdout:1046: done (2 left) 2024-04-24T01:59:25.310 INFO:tasks.rados.rados.0.smithi079.stdout:1047: done (1 left) 2024-04-24T01:59:25.310 INFO:tasks.rados.rados.0.smithi079.stdout:1048: done (0 left) 2024-04-24T01:59:25.310 INFO:tasks.rados.rados.0.smithi079.stdout:1049: read oid 22 snap -1 2024-04-24T01:59:25.310 INFO:tasks.rados.rados.0.smithi079.stdout:1049: expect (ObjNum 435 snap 99 seq_num 435) 2024-04-24T01:59:25.310 INFO:tasks.rados.rados.0.smithi079.stdout:1050: write oid 16 current snap is 116 2024-04-24T01:59:25.310 INFO:tasks.rados.rados.0.smithi079.stdout:1050: seq_num 522 ranges {773254=791982,2346424=426830} 2024-04-24T01:59:25.317 INFO:tasks.rados.rados.0.smithi079.stdout:1050: writing smithi07929762-16 from 773254 to 1565236 tid 1 2024-04-24T01:59:25.326 INFO:tasks.rados.rados.0.smithi079.stdout:1050: writing smithi07929762-16 from 2346424 to 2773254 tid 2 2024-04-24T01:59:25.326 INFO:tasks.rados.rados.0.smithi079.stdout:1051: read oid 7 snap 114 2024-04-24T01:59:25.326 INFO:tasks.rados.rados.0.smithi079.stdout:1051: expect (ObjNum 375 snap 90 seq_num 375) 2024-04-24T01:59:25.326 INFO:tasks.rados.rados.0.smithi079.stdout:1052: delete oid 10 current snap is 116 2024-04-24T01:59:25.336 INFO:tasks.rados.rados.0.smithi079.stdout:1049: expect (ObjNum 435 snap 99 seq_num 435) 2024-04-24T01:59:25.526 INFO:tasks.rados.rados.0.smithi079.stdout:1051: expect (ObjNum 375 snap 90 seq_num 375) 2024-04-24T01:59:25.613 INFO:tasks.rados.rados.0.smithi079.stdout:1050: finishing write tid 1 to smithi07929762-16 2024-04-24T01:59:25.614 INFO:tasks.rados.rados.0.smithi079.stdout:1050: finishing write tid 2 to smithi07929762-16 2024-04-24T01:59:25.614 INFO:tasks.rados.rados.0.smithi079.stdout:1050: finishing write tid 3 to smithi07929762-16 2024-04-24T01:59:25.614 INFO:tasks.rados.rados.0.smithi079.stdout:1050: finishing write tid 4 to smithi07929762-16 2024-04-24T01:59:25.614 INFO:tasks.rados.rados.0.smithi079.stdout:1050: oid 16 updating version 0 to 483 2024-04-24T01:59:25.614 INFO:tasks.rados.rados.0.smithi079.stdout:1050: oid 16 version 483 is already newer than 483 2024-04-24T01:59:25.614 INFO:tasks.rados.rados.0.smithi079.stdout:1050: oid 16 version 483 is already newer than 483 2024-04-24T01:59:25.614 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 16 v 483 (ObjNum 522 snap 116 seq_num 522) dirty exists 2024-04-24T01:59:25.614 INFO:tasks.rados.rados.0.smithi079.stdout:1050: left oid 16 (ObjNum 522 snap 116 seq_num 522) 2024-04-24T01:59:25.614 INFO:tasks.rados.rados.0.smithi079.stdout:1049: done (3 left) 2024-04-24T01:59:25.614 INFO:tasks.rados.rados.0.smithi079.stdout:1050: done (2 left) 2024-04-24T01:59:25.614 INFO:tasks.rados.rados.0.smithi079.stdout:1051: done (1 left) 2024-04-24T01:59:25.614 INFO:tasks.rados.rados.0.smithi079.stdout:1052: done (0 left) 2024-04-24T01:59:25.615 INFO:tasks.rados.rados.0.smithi079.stdout:1053: snap_create 2024-04-24T01:59:26.071 INFO:tasks.rados.rados.0.smithi079.stdout:1053: done (0 left) 2024-04-24T01:59:26.071 INFO:tasks.rados.rados.0.smithi079.stdout:1054: write (excl) oid 38 current snap is 117 2024-04-24T01:59:26.071 INFO:tasks.rados.rados.0.smithi079.stdout:1054: seq_num 524 ranges {406868=542480,1547035=401512,2690354=116514} 2024-04-24T01:59:26.075 INFO:tasks.rados.rados.0.smithi079.stdout:1054: writing smithi07929762-38 from 406868 to 949348 tid 1 2024-04-24T01:59:26.081 INFO:tasks.rados.rados.0.smithi079.stdout:1054: writing smithi07929762-38 from 1547035 to 1948547 tid 2 2024-04-24T01:59:26.086 INFO:tasks.rados.rados.0.smithi079.stdout:1054: writing smithi07929762-38 from 2690354 to 2806868 tid 3 2024-04-24T01:59:26.086 INFO:tasks.rados.rados.0.smithi079.stdout:1055: read oid 25 snap -1 2024-04-24T01:59:26.086 INFO:tasks.rados.rados.0.smithi079.stdout:1055: expect (ObjNum 508 snap 113 seq_num 508) 2024-04-24T01:59:26.086 INFO:tasks.rados.rados.0.smithi079.stdout:1056: read oid 17 snap -1 2024-04-24T01:59:26.086 INFO:tasks.rados.rados.0.smithi079.stdout:1056: expect (ObjNum 515 snap 115 seq_num 515) 2024-04-24T01:59:26.086 INFO:tasks.rados.rados.0.smithi079.stdout:1057: read oid 43 snap -1 2024-04-24T01:59:26.086 INFO:tasks.rados.rados.0.smithi079.stdout:1057: expect deleted 2024-04-24T01:59:26.086 INFO:tasks.rados.rados.0.smithi079.stdout:1058: read oid 18 snap 115 2024-04-24T01:59:26.087 INFO:tasks.rados.rados.0.smithi079.stdout:1058: expect (ObjNum 477 snap 109 seq_num 477) 2024-04-24T01:59:26.087 INFO:tasks.rados.rados.0.smithi079.stdout:1059: snap_create 2024-04-24T01:59:26.137 INFO:tasks.rados.rados.0.smithi079.stdout:1055: expect (ObjNum 508 snap 113 seq_num 508) 2024-04-24T01:59:26.304 INFO:tasks.rados.rados.0.smithi079.stdout:1058: expect (ObjNum 477 snap 109 seq_num 477) 2024-04-24T01:59:26.381 INFO:tasks.rados.rados.0.smithi079.stdout:1054: finishing write tid 1 to smithi07929762-38 2024-04-24T01:59:26.381 INFO:tasks.rados.rados.0.smithi079.stdout:1054: finishing write tid 2 to smithi07929762-38 2024-04-24T01:59:26.381 INFO:tasks.rados.rados.0.smithi079.stdout:1054: finishing write tid 3 to smithi07929762-38 2024-04-24T01:59:26.382 INFO:tasks.rados.rados.0.smithi079.stdout:1054: finishing write tid 4 to smithi07929762-38 2024-04-24T01:59:26.382 INFO:tasks.rados.rados.0.smithi079.stdout:1054: finishing write tid 5 to smithi07929762-38 2024-04-24T01:59:26.382 INFO:tasks.rados.rados.0.smithi079.stdout:1054: oid 38 updating version 0 to 244 2024-04-24T01:59:26.382 INFO:tasks.rados.rados.0.smithi079.stdout:1054: oid 38 version 244 is already newer than 241 2024-04-24T01:59:26.382 INFO:tasks.rados.rados.0.smithi079.stdout:1054: oid 38 version 244 is already newer than 244 2024-04-24T01:59:26.382 INFO:tasks.rados.rados.0.smithi079.stdout:1054: oid 38 version 244 is already newer than 244 2024-04-24T01:59:26.382 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 38 v 244 (ObjNum 524 snap 117 seq_num 524) dirty exists 2024-04-24T01:59:26.382 INFO:tasks.rados.rados.0.smithi079.stdout:1054: left oid 38 (ObjNum 524 snap 117 seq_num 524) 2024-04-24T01:59:26.382 INFO:tasks.rados.rados.0.smithi079.stdout:1056: expect (ObjNum 515 snap 115 seq_num 515) 2024-04-24T01:59:26.741 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T01:59:26.848 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:59:26.845+0000 7efc1b95d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:59:26.851 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:59:26.850+0000 7efc1b95d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:59:27.076 INFO:tasks.rados.rados.0.smithi079.stdout:1054: done (5 left) 2024-04-24T01:59:27.076 INFO:tasks.rados.rados.0.smithi079.stdout:1055: done (4 left) 2024-04-24T01:59:27.076 INFO:tasks.rados.rados.0.smithi079.stdout:1056: done (3 left) 2024-04-24T01:59:27.076 INFO:tasks.rados.rados.0.smithi079.stdout:1057: done (2 left) 2024-04-24T01:59:27.076 INFO:tasks.rados.rados.0.smithi079.stdout:1058: done (1 left) 2024-04-24T01:59:27.076 INFO:tasks.rados.rados.0.smithi079.stdout:1059: done (0 left) 2024-04-24T01:59:27.076 INFO:tasks.rados.rados.0.smithi079.stdout:1060: snap_create 2024-04-24T01:59:28.078 INFO:tasks.rados.rados.0.smithi079.stdout:1060: done (0 left) 2024-04-24T01:59:28.097 INFO:tasks.rados.rados.0.smithi079.stdout:1061: read oid 24 snap 114 2024-04-24T01:59:28.097 INFO:tasks.rados.rados.0.smithi079.stdout:1061: expect deleted 2024-04-24T01:59:28.097 INFO:tasks.rados.rados.0.smithi079.stdout:1062: snap_create 2024-04-24T01:59:28.946 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T01:59:29.063 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:59:29.060+0000 7fb8894ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:59:29.067 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:59:29.066+0000 7fb8894ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:59:29.084 INFO:tasks.rados.rados.0.smithi079.stdout:1061: done (1 left) 2024-04-24T01:59:29.084 INFO:tasks.rados.rados.0.smithi079.stdout:1062: done (0 left) 2024-04-24T01:59:29.084 INFO:tasks.rados.rados.0.smithi079.stdout:1063: write (excl) oid 37 current snap is 120 2024-04-24T01:59:29.084 INFO:tasks.rados.rados.0.smithi079.stdout:1063: seq_num 525 ranges {423675=734082,1721870=545170,2717777=105898} 2024-04-24T01:59:29.090 INFO:tasks.rados.rados.0.smithi079.stdout:1063: writing smithi07929762-37 from 423675 to 1157757 tid 1 2024-04-24T01:59:29.096 INFO:tasks.rados.rados.0.smithi079.stdout:1063: writing smithi07929762-37 from 1721870 to 2267040 tid 2 2024-04-24T01:59:29.101 INFO:tasks.rados.rados.0.smithi079.stdout:1063: writing smithi07929762-37 from 2717777 to 2823675 tid 3 2024-04-24T01:59:29.101 INFO:tasks.rados.rados.0.smithi079.stdout:1064: read oid 22 snap -1 2024-04-24T01:59:29.101 INFO:tasks.rados.rados.0.smithi079.stdout:1064: expect (ObjNum 435 snap 99 seq_num 435) 2024-04-24T01:59:29.101 INFO:tasks.rados.rados.0.smithi079.stdout:1065: snap_remove snap 119 2024-04-24T01:59:29.932 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-04-24T01:59:29.932 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-04-24T01:59:29.932 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T01:59:30.043 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:59:30.039+0000 7f36fa5ff640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:59:30.048 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:59:30.046+0000 7f36f939d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:59:30.089 INFO:tasks.rados.rados.0.smithi079.stdout:1065: done (2 left) 2024-04-24T01:59:30.089 INFO:tasks.rados.rados.0.smithi079.stdout:1066: read oid 15 snap -1 2024-04-24T01:59:30.089 INFO:tasks.rados.rados.0.smithi079.stdout:1066: expect (ObjNum 437 snap 99 seq_num 437) 2024-04-24T01:59:30.089 INFO:tasks.rados.rados.0.smithi079.stdout:1067: write (excl) oid 16 current snap is 120 2024-04-24T01:59:30.089 INFO:tasks.rados.rados.0.smithi079.stdout:1067: seq_num 526 ranges {440482=409331,1380352=405181,2461553=378929} 2024-04-24T01:59:30.093 INFO:tasks.rados.rados.0.smithi079.stdout:1067: writing smithi07929762-16 from 440482 to 849813 tid 1 2024-04-24T01:59:30.100 INFO:tasks.rados.rados.0.smithi079.stdout:1067: writing smithi07929762-16 from 1380352 to 1785533 tid 2 2024-04-24T01:59:30.108 INFO:tasks.rados.rados.0.smithi079.stdout:1067: writing smithi07929762-16 from 2461553 to 2840482 tid 3 2024-04-24T01:59:30.108 INFO:tasks.rados.rados.0.smithi079.stdout:1068: read oid 6 snap -1 2024-04-24T01:59:30.108 INFO:tasks.rados.rados.0.smithi079.stdout:1068: expect (ObjNum 276 snap 71 seq_num 276) 2024-04-24T01:59:30.108 INFO:tasks.rados.rados.0.smithi079.stdout:1069: write (excl) oid 30 current snap is 120 2024-04-24T01:59:30.108 INFO:tasks.rados.rados.0.smithi079.stdout:1069: seq_num 527 ranges {457289=484580,1438834=548839,2772623=84666} 2024-04-24T01:59:30.113 INFO:tasks.rados.rados.0.smithi079.stdout:1069: writing smithi07929762-30 from 457289 to 941869 tid 1 2024-04-24T01:59:30.120 INFO:tasks.rados.rados.0.smithi079.stdout:1069: writing smithi07929762-30 from 1438834 to 1987673 tid 2 2024-04-24T01:59:30.126 INFO:tasks.rados.rados.0.smithi079.stdout:1069: writing smithi07929762-30 from 2772623 to 2857289 tid 3 2024-04-24T01:59:30.126 INFO:tasks.rados.rados.0.smithi079.stdout:1070: write oid 23 current snap is 120 2024-04-24T01:59:30.126 INFO:tasks.rados.rados.0.smithi079.stdout:1070: seq_num 528 ranges {474096=559829,1780963=408850,2800046=74050} 2024-04-24T01:59:30.134 INFO:tasks.rados.rados.0.smithi079.stdout:1070: writing smithi07929762-23 from 474096 to 1033925 tid 1 2024-04-24T01:59:30.142 INFO:tasks.rados.rados.0.smithi079.stdout:1070: writing smithi07929762-23 from 1780963 to 2189813 tid 2 2024-04-24T01:59:30.146 INFO:tasks.rados.rados.0.smithi079.stdout:1070: writing smithi07929762-23 from 2800046 to 2874096 tid 3 2024-04-24T01:59:30.146 INFO:tasks.rados.rados.0.smithi079.stdout:1071: write oid 43 current snap is 120 2024-04-24T01:59:30.146 INFO:tasks.rados.rados.0.smithi079.stdout:1064: expect (ObjNum 435 snap 99 seq_num 435) 2024-04-24T01:59:30.320 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:59:30.320 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-24T01:59:30.342 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":372,"stamp":"2024-04-24T01:59:30.087401+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":78358898,"num_objects":44,"num_object_clones":9,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":2014,"num_read_kb":1354495,"num_write":1861,"num_write_kb":470564,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3539,"ondisk_log_size":3539,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":316428,"kb_used_data":175852,"kb_used_omap":0,"kb_used_meta":140544,"kb_avail":377170932,"statfs":{"total":386547056640,"available":386223034368,"internally_reserved":0,"allocated":180072448,"data_stored":181003495,"data_compressed":89889,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":143917056},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-3932805,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":60,"num_read_kb":54574,"num_write":28,"num_write_kb":7275,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.017810"},"pg_stats":[{"pgid":"3.f","version":"247'181","reported_seq":939,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-24T01:59:29.128759+0000","last_change":"2024-04-24T01:59:24.132031+0000","last_active":"2024-04-24T01:59:29.128759+0000","last_peered":"2024-04-24T01:59:29.128759+0000","last_clean":"2024-04-24T01:59:29.128759+0000","last_became_active":"2024-04-24T01:55:30.386726+0000","last_became_peered":"2024-04-24T01:55:30.386726+0000","last_unstale":"2024-04-24T01:59:29.128759+0000","last_undegraded":"2024-04-24T01:59:29.128759+0000","last_fullsized":"2024-04-24T01:59:29.128759+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":181,"log_dups_size":0,"ondisk_log_size":181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":17,"snaptrim_duration":0,"stat_sum":{"num_bytes":4874195,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":134,"num_read_kb":119395,"num_write":88,"num_write_kb":25231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"248'301","reported_seq":1001,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-04-24T01:59:26.120098+0000","last_change":"2024-04-24T01:59:24.088715+0000","last_active":"2024-04-24T01:59:26.120098+0000","last_peered":"2024-04-24T01:59:26.120098+0000","last_clean":"2024-04-24T01:59:26.120098+0000","last_became_active":"2024-04-24T01:55:30.321866+0000","last_became_peered":"2024-04-24T01:55:30.321866+0000","last_unstale":"2024-04-24T01:59:26.120098+0000","last_undegraded":"2024-04-24T01:59:26.120098+0000","last_fullsized":"2024-04-24T01:59:26.120098+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":301,"log_dups_size":0,"ondisk_log_size":301,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":27,"snaptrim_duration":0,"stat_sum":{"num_bytes":5143140,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":165,"num_read_kb":113141,"num_write":151,"num_write_kb":40824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"247'225","reported_seq":940,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-04-24T01:59:26.099574+0000","last_change":"2024-04-24T01:59:24.091091+0000","last_active":"2024-04-24T01:59:26.099574+0000","last_peered":"2024-04-24T01:59:26.099574+0000","last_clean":"2024-04-24T01:59:26.099574+0000","last_became_active":"2024-04-24T01:55:30.320186+0000","last_became_peered":"2024-04-24T01:55:30.320186+0000","last_unstale":"2024-04-24T01:59:26.099574+0000","last_undegraded":"2024-04-24T01:59:26.099574+0000","last_fullsized":"2024-04-24T01:59:26.099574+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":225,"log_dups_size":0,"ondisk_log_size":225,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":22,"snaptrim_duration":0,"stat_sum":{"num_bytes":5159914,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":119,"num_read_kb":79388,"num_write":124,"num_write_kb":34018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"246'263","reported_seq":995,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-24T01:59:29.119492+0000","last_change":"2024-04-24T01:59:24.096069+0000","last_active":"2024-04-24T01:59:29.119492+0000","last_peered":"2024-04-24T01:59:29.119492+0000","last_clean":"2024-04-24T01:59:29.119492+0000","last_became_active":"2024-04-24T01:55:30.288081+0000","last_became_peered":"2024-04-24T01:55:30.288081+0000","last_unstale":"2024-04-24T01:59:29.119492+0000","last_undegraded":"2024-04-24T01:59:29.119492+0000","last_fullsized":"2024-04-24T01:59:29.119492+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":263,"log_dups_size":0,"ondisk_log_size":263,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":27,"snaptrim_duration":0,"stat_sum":{"num_bytes":2269044,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":136,"num_read_kb":92553,"num_write":129,"num_write_kb":33953,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"247'91","reported_seq":823,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-24T01:59:29.109885+0000","last_change":"2024-04-24T01:59:25.100353+0000","last_active":"2024-04-24T01:59:29.109885+0000","last_peered":"2024-04-24T01:59:29.109885+0000","last_clean":"2024-04-24T01:59:29.109885+0000","last_became_active":"2024-04-24T01:55:30.333815+0000","last_became_peered":"2024-04-24T01:55:30.333815+0000","last_unstale":"2024-04-24T01:59:29.109885+0000","last_undegraded":"2024-04-24T01:59:29.109885+0000","last_fullsized":"2024-04-24T01:59:29.109885+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":10,"snaptrim_duration":0,"stat_sum":{"num_bytes":2319465,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":44,"num_read_kb":23146,"num_write":55,"num_write_kb":15361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"252'486","reported_seq":1183,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-24T01:59:29.173513+0000","last_change":"2024-04-24T01:59:24.173881+0000","last_active":"2024-04-24T01:59:29.173513+0000","last_peered":"2024-04-24T01:59:29.173513+0000","last_clean":"2024-04-24T01:59:29.173513+0000","last_became_active":"2024-04-24T01:55:30.315075+0000","last_became_peered":"2024-04-24T01:55:30.315075+0000","last_unstale":"2024-04-24T01:59:29.173513+0000","last_undegraded":"2024-04-24T01:59:29.173513+0000","last_fullsized":"2024-04-24T01:59:29.173513+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":486,"log_dups_size":0,"ondisk_log_size":486,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":46,"snaptrim_duration":0,"stat_sum":{"num_bytes":12370447,"num_objects":7,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":232,"num_read_kb":146267,"num_write":224,"num_write_kb":59453,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"244'81","reported_seq":834,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-24T01:59:29.134811+0000","last_change":"2024-04-24T01:59:24.103404+0000","last_active":"2024-04-24T01:59:29.134811+0000","last_peered":"2024-04-24T01:59:29.134811+0000","last_clean":"2024-04-24T01:59:29.134811+0000","last_became_active":"2024-04-24T01:55:30.350300+0000","last_became_peered":"2024-04-24T01:55:30.350300+0000","last_unstale":"2024-04-24T01:59:29.134811+0000","last_undegraded":"2024-04-24T01:59:29.134811+0000","last_fullsized":"2024-04-24T01:59:29.134811+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":8,"snaptrim_duration":0,"stat_sum":{"num_bytes":6487667,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":44,"num_read_kb":22973,"num_write":56,"num_write_kb":14433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"247'26","reported_seq":754,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-04-24T01:59:26.108388+0000","last_change":"2024-04-24T01:59:24.121651+0000","last_active":"2024-04-24T01:59:26.108388+0000","last_peered":"2024-04-24T01:59:26.108388+0000","last_clean":"2024-04-24T01:59:26.108388+0000","last_became_active":"2024-04-24T01:55:30.376813+0000","last_became_peered":"2024-04-24T01:55:30.376813+0000","last_unstale":"2024-04-24T01:59:26.108388+0000","last_undegraded":"2024-04-24T01:59:26.108388+0000","last_fullsized":"2024-04-24T01:59:26.108388+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":14,"num_read_kb":10081,"num_write":22,"num_write_kb":5614,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'97","reported_seq":766,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-24T01:59:29.094609+0000","last_change":"2024-04-24T01:55:16.196470+0000","last_active":"2024-04-24T01:59:29.094609+0000","last_peered":"2024-04-24T01:59:29.094609+0000","last_clean":"2024-04-24T01:59:29.094609+0000","last_became_active":"2024-04-24T01:55:16.190464+0000","last_became_peered":"2024-04-24T01:55:16.190464+0000","last_unstale":"2024-04-24T01:59:29.094609+0000","last_undegraded":"2024-04-24T01:59:29.094609+0000","last_fullsized":"2024-04-24T01:59:29.094609+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:15.127115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:15.127115+0000","last_clean_scrub_stamp":"2024-04-24T01:55:15.127115+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":624,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-04-24T01:59:26.113706+0000","last_change":"2024-04-24T01:55:18.236148+0000","last_active":"2024-04-24T01:59:26.113706+0000","last_peered":"2024-04-24T01:59:26.113706+0000","last_clean":"2024-04-24T01:59:26.113706+0000","last_became_active":"2024-04-24T01:55:18.230154+0000","last_became_peered":"2024-04-24T01:55:18.230154+0000","last_unstale":"2024-04-24T01:59:26.113706+0000","last_undegraded":"2024-04-24T01:59:26.113706+0000","last_fullsized":"2024-04-24T01:59:26.113706+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"247'369","reported_seq":1087,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-04-24T01:59:26.133117+0000","last_change":"2024-04-24T01:59:24.196186+0000","last_active":"2024-04-24T01:59:26.133117+0000","last_peered":"2024-04-24T01:59:26.133117+0000","last_clean":"2024-04-24T01:59:26.133117+0000","last_became_active":"2024-04-24T01:55:30.316222+0000","last_became_peered":"2024-04-24T01:55:30.316222+0000","last_unstale":"2024-04-24T01:59:26.133117+0000","last_undegraded":"2024-04-24T01:59:26.133117+0000","last_fullsized":"2024-04-24T01:59:26.133117+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":369,"log_dups_size":0,"ondisk_log_size":369,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":29,"snaptrim_duration":0,"stat_sum":{"num_bytes":8924781,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":203,"num_read_kb":143511,"num_write":195,"num_write_kb":54798,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":647,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-24T01:59:29.136503+0000","last_change":"2024-04-24T01:55:18.265905+0000","last_active":"2024-04-24T01:59:29.136503+0000","last_peered":"2024-04-24T01:59:29.136503+0000","last_clean":"2024-04-24T01:59:29.136503+0000","last_became_active":"2024-04-24T01:55:18.257020+0000","last_became_peered":"2024-04-24T01:55:18.257020+0000","last_unstale":"2024-04-24T01:59:29.136503+0000","last_undegraded":"2024-04-24T01:59:29.136503+0000","last_fullsized":"2024-04-24T01:59:29.136503+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"247'305","reported_seq":1002,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-24T01:59:29.098748+0000","last_change":"2024-04-24T01:59:25.091151+0000","last_active":"2024-04-24T01:59:29.098748+0000","last_peered":"2024-04-24T01:59:29.098748+0000","last_clean":"2024-04-24T01:59:29.098748+0000","last_became_active":"2024-04-24T01:55:30.344129+0000","last_became_peered":"2024-04-24T01:55:30.344129+0000","last_unstale":"2024-04-24T01:59:29.098748+0000","last_undegraded":"2024-04-24T01:59:29.098748+0000","last_fullsized":"2024-04-24T01:59:29.098748+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":305,"log_dups_size":0,"ondisk_log_size":305,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":26,"snaptrim_duration":0,"stat_sum":{"num_bytes":2706026,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":129,"num_read_kb":72537,"num_write":140,"num_write_kb":39093,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":624,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-04-24T01:59:26.100772+0000","last_change":"2024-04-24T01:55:18.215251+0000","last_active":"2024-04-24T01:59:26.100772+0000","last_peered":"2024-04-24T01:59:26.100772+0000","last_clean":"2024-04-24T01:59:26.100772+0000","last_became_active":"2024-04-24T01:55:18.206605+0000","last_became_peered":"2024-04-24T01:55:18.206605+0000","last_unstale":"2024-04-24T01:59:26.100772+0000","last_undegraded":"2024-04-24T01:59:26.100772+0000","last_fullsized":"2024-04-24T01:59:26.100772+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"247'144","reported_seq":894,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-24T01:59:29.124750+0000","last_change":"2024-04-24T01:59:25.095459+0000","last_active":"2024-04-24T01:59:29.124750+0000","last_peered":"2024-04-24T01:59:29.124750+0000","last_clean":"2024-04-24T01:59:29.124750+0000","last_became_active":"2024-04-24T01:55:30.401366+0000","last_became_peered":"2024-04-24T01:55:30.401366+0000","last_unstale":"2024-04-24T01:59:29.124750+0000","last_undegraded":"2024-04-24T01:59:29.124750+0000","last_fullsized":"2024-04-24T01:59:29.124750+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":144,"log_dups_size":0,"ondisk_log_size":144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":11,"snaptrim_duration":0,"stat_sum":{"num_bytes":5563282,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":130,"num_read_kb":104968,"num_write":55,"num_write_kb":14247,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":647,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-24T01:59:29.136525+0000","last_change":"2024-04-24T01:55:18.230893+0000","last_active":"2024-04-24T01:59:29.136525+0000","last_peered":"2024-04-24T01:59:29.136525+0000","last_clean":"2024-04-24T01:59:29.136525+0000","last_became_active":"2024-04-24T01:55:18.225188+0000","last_became_peered":"2024-04-24T01:55:18.225188+0000","last_unstale":"2024-04-24T01:59:29.136525+0000","last_undegraded":"2024-04-24T01:59:29.136525+0000","last_fullsized":"2024-04-24T01:59:29.136525+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"246'242","reported_seq":934,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-04-24T01:59:26.110158+0000","last_change":"2024-04-24T01:59:24.089437+0000","last_active":"2024-04-24T01:59:26.110158+0000","last_peered":"2024-04-24T01:59:26.110158+0000","last_clean":"2024-04-24T01:59:26.110158+0000","last_became_active":"2024-04-24T01:55:30.343318+0000","last_became_peered":"2024-04-24T01:55:30.343318+0000","last_unstale":"2024-04-24T01:59:26.110158+0000","last_undegraded":"2024-04-24T01:59:26.110158+0000","last_fullsized":"2024-04-24T01:59:26.110158+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":242,"log_dups_size":0,"ondisk_log_size":242,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":23,"snaptrim_duration":0,"stat_sum":{"num_bytes":2302625,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":126,"num_read_kb":103525,"num_write":108,"num_write_kb":30307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"248'318","reported_seq":1023,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-04-24T01:59:26.112399+0000","last_change":"2024-04-24T01:59:24.120583+0000","last_active":"2024-04-24T01:59:26.112399+0000","last_peered":"2024-04-24T01:59:26.112399+0000","last_clean":"2024-04-24T01:59:26.112399+0000","last_became_active":"2024-04-24T01:55:30.345308+0000","last_became_peered":"2024-04-24T01:55:30.345308+0000","last_unstale":"2024-04-24T01:59:26.112399+0000","last_undegraded":"2024-04-24T01:59:26.112399+0000","last_fullsized":"2024-04-24T01:59:26.112399+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":318,"log_dups_size":0,"ondisk_log_size":318,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":27,"snaptrim_duration":0,"stat_sum":{"num_bytes":2168202,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":167,"num_read_kb":110065,"num_write":149,"num_write_kb":39126,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":610,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-04-24T01:59:26.126059+0000","last_change":"2024-04-24T01:55:18.224238+0000","last_active":"2024-04-24T01:59:26.126059+0000","last_peered":"2024-04-24T01:59:26.126059+0000","last_clean":"2024-04-24T01:59:26.126059+0000","last_became_active":"2024-04-24T01:55:18.219131+0000","last_became_peered":"2024-04-24T01:55:18.219131+0000","last_unstale":"2024-04-24T01:59:26.126059+0000","last_undegraded":"2024-04-24T01:59:26.126059+0000","last_fullsized":"2024-04-24T01:59:26.126059+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"217'58","reported_seq":778,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-04-24T01:59:26.119449+0000","last_change":"2024-04-24T01:59:24.094413+0000","last_active":"2024-04-24T01:59:26.119449+0000","last_peered":"2024-04-24T01:59:26.119449+0000","last_clean":"2024-04-24T01:59:26.119449+0000","last_became_active":"2024-04-24T01:55:30.319958+0000","last_became_peered":"2024-04-24T01:55:30.319958+0000","last_unstale":"2024-04-24T01:59:26.119449+0000","last_undegraded":"2024-04-24T01:59:26.119449+0000","last_fullsized":"2024-04-24T01:59:26.119449+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":5,"snaptrim_duration":0,"stat_sum":{"num_bytes":3428694,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":39,"num_read_kb":36472,"num_write":42,"num_write_kb":11889,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":633,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-24T01:59:29.105703+0000","last_change":"2024-04-24T01:55:18.235763+0000","last_active":"2024-04-24T01:59:29.105703+0000","last_peered":"2024-04-24T01:59:29.105703+0000","last_clean":"2024-04-24T01:59:29.105703+0000","last_became_active":"2024-04-24T01:55:18.229175+0000","last_became_peered":"2024-04-24T01:55:18.229175+0000","last_unstale":"2024-04-24T01:59:29.105703+0000","last_undegraded":"2024-04-24T01:59:29.105703+0000","last_fullsized":"2024-04-24T01:59:29.105703+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"247'108","reported_seq":834,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-24T01:59:29.096525+0000","last_change":"2024-04-24T01:59:25.091406+0000","last_active":"2024-04-24T01:59:29.096525+0000","last_peered":"2024-04-24T01:59:29.096525+0000","last_clean":"2024-04-24T01:59:29.096525+0000","last_became_active":"2024-04-24T01:55:30.372690+0000","last_became_peered":"2024-04-24T01:55:30.372690+0000","last_unstale":"2024-04-24T01:59:29.096525+0000","last_undegraded":"2024-04-24T01:59:29.096525+0000","last_fullsized":"2024-04-24T01:59:29.096525+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":9,"snaptrim_duration":0,"stat_sum":{"num_bytes":2605184,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":55,"num_read_kb":39372,"num_write":49,"num_write_kb":14242,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":610,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-04-24T01:59:26.117503+0000","last_change":"2024-04-24T01:55:18.278351+0000","last_active":"2024-04-24T01:59:26.117503+0000","last_peered":"2024-04-24T01:59:26.117503+0000","last_clean":"2024-04-24T01:59:26.117503+0000","last_became_active":"2024-04-24T01:55:18.273668+0000","last_became_peered":"2024-04-24T01:55:18.273668+0000","last_unstale":"2024-04-24T01:59:26.117503+0000","last_undegraded":"2024-04-24T01:59:26.117503+0000","last_fullsized":"2024-04-24T01:59:26.117503+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"249'244","reported_seq":970,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-04-24T01:59:26.193627+0000","last_change":"2024-04-24T01:59:24.094930+0000","last_active":"2024-04-24T01:59:26.193627+0000","last_peered":"2024-04-24T01:59:26.193627+0000","last_clean":"2024-04-24T01:59:26.193627+0000","last_became_active":"2024-04-24T01:55:30.363123+0000","last_became_peered":"2024-04-24T01:55:30.363123+0000","last_unstale":"2024-04-24T01:59:26.193627+0000","last_undegraded":"2024-04-24T01:59:26.193627+0000","last_fullsized":"2024-04-24T01:59:26.193627+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":244,"log_dups_size":0,"ondisk_log_size":244,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":27,"snaptrim_duration":0,"stat_sum":{"num_bytes":11445864,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":174,"num_read_kb":137010,"num_write":137,"num_write_kb":36081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":624,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-04-24T01:59:26.110786+0000","last_change":"2024-04-24T01:55:18.241117+0000","last_active":"2024-04-24T01:59:26.110786+0000","last_peered":"2024-04-24T01:59:26.110786+0000","last_clean":"2024-04-24T01:59:26.110786+0000","last_became_active":"2024-04-24T01:55:18.234313+0000","last_became_peered":"2024-04-24T01:55:18.234313+0000","last_unstale":"2024-04-24T01:59:26.110786+0000","last_undegraded":"2024-04-24T01:59:26.110786+0000","last_fullsized":"2024-04-24T01:59:26.110786+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":77768530,"num_objects":42,"num_object_clones":9,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":1911,"num_read_kb":1354404,"num_write":1724,"num_write_kb":468670,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":173821952,"data_stored":179191716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3442,"ondisk_log_size":3442,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":1}],"osd_stats":[{"osd":3,"up_from":10,"seq":42949673214,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":83200,"kb_used_data":48056,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94288640,"statfs":{"total":96636764160,"available":96551567360,"internally_reserved":0,"allocated":49209344,"data_stored":49105906,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673214,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":74372,"kb_used_data":39228,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94297468,"statfs":{"total":96636764160,"available":96560607232,"internally_reserved":0,"allocated":40169472,"data_stored":40448522,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705919,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":72184,"kb_used_data":37040,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94299656,"statfs":{"total":96636764160,"available":96562847744,"internally_reserved":0,"allocated":37928960,"data_stored":38422372,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705918,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":86672,"kb_used_data":51528,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94285168,"statfs":{"total":96636764160,"available":96548012032,"internally_reserved":0,"allocated":52764672,"data_stored":53026695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":37212160,"data_stored":38789780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":50720768,"data_stored":52137158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38801408,"data_stored":40645987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":47087616,"data_stored":47618791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T01:59:30.360 INFO:tasks.rados.rados.0.smithi079.stdout:1063: finishing write tid 1 to smithi07929762-37 2024-04-24T01:59:30.360 INFO:tasks.rados.rados.0.smithi079.stdout:1063: finishing write tid 2 to smithi07929762-37 2024-04-24T01:59:30.360 INFO:tasks.rados.rados.0.smithi079.stdout:1063: finishing write tid 3 to smithi07929762-37 2024-04-24T01:59:30.360 INFO:tasks.rados.rados.0.smithi079.stdout:1063: finishing write tid 4 to smithi07929762-37 2024-04-24T01:59:30.360 INFO:tasks.rados.rados.0.smithi079.stdout:1063: finishing write tid 5 to smithi07929762-37 2024-04-24T01:59:30.360 INFO:tasks.rados.rados.0.smithi079.stdout:1063: oid 37 updating version 0 to 488 2024-04-24T01:59:30.360 INFO:tasks.rados.rados.0.smithi079.stdout:1063: oid 37 version 488 is already newer than 485 2024-04-24T01:59:30.360 INFO:tasks.rados.rados.0.smithi079.stdout:1063: oid 37 version 488 is already newer than 488 2024-04-24T01:59:30.360 INFO:tasks.rados.rados.0.smithi079.stdout:1063: oid 37 version 488 is already newer than 488 2024-04-24T01:59:30.360 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 37 v 488 (ObjNum 525 snap 120 seq_num 525) dirty exists 2024-04-24T01:59:30.360 INFO:tasks.rados.rados.0.smithi079.stdout:1063: left oid 37 (ObjNum 525 snap 120 seq_num 525) 2024-04-24T01:59:30.361 INFO:tasks.rados.rados.0.smithi079.stdout:1071: seq_num 529 ranges {490903=635078,1839445=552508} 2024-04-24T01:59:30.368 INFO:tasks.rados.rados.0.smithi079.stdout:1071: writing smithi07929762-43 from 490903 to 1125981 tid 1 2024-04-24T01:59:30.376 INFO:tasks.rados.rados.0.smithi079.stdout:1071: writing smithi07929762-43 from 1839445 to 2391953 tid 2 2024-04-24T01:59:30.376 INFO:tasks.rados.rados.0.smithi079.stdout:1063: done (7 left) 2024-04-24T01:59:30.376 INFO:tasks.rados.rados.0.smithi079.stdout:1064: done (6 left) 2024-04-24T01:59:30.376 INFO:tasks.rados.rados.0.smithi079.stdout:1072: write oid 47 current snap is 120 2024-04-24T01:59:30.376 INFO:tasks.rados.rados.0.smithi079.stdout:1072: seq_num 530 ranges {507710=710327,1897927=412519,2854892=52818} 2024-04-24T01:59:30.384 INFO:tasks.rados.rados.0.smithi079.stdout:1072: writing smithi07929762-47 from 507710 to 1218037 tid 1 2024-04-24T01:59:30.392 INFO:tasks.rados.rados.0.smithi079.stdout:1072: writing smithi07929762-47 from 1897927 to 2310446 tid 2 2024-04-24T01:59:30.395 INFO:tasks.rados.rados.0.smithi079.stdout:1072: writing smithi07929762-47 from 2854892 to 2907710 tid 3 2024-04-24T01:59:30.395 INFO:tasks.rados.rados.0.smithi079.stdout:1073: snap_remove snap 116 2024-04-24T01:59:31.049 INFO:tasks.rados.rados.0.smithi079.stdout:1073: done (7 left) 2024-04-24T01:59:31.049 INFO:tasks.rados.rados.0.smithi079.stdout:1074: write oid 9 current snap is 120 2024-04-24T01:59:31.049 INFO:tasks.rados.rados.0.smithi079.stdout:1074: seq_num 531 ranges {524517=785576,1956409=556177} 2024-04-24T01:59:31.056 INFO:tasks.rados.rados.0.smithi079.stdout:1074: writing smithi07929762-9 from 524517 to 1310093 tid 1 2024-04-24T01:59:31.064 INFO:tasks.rados.rados.0.smithi079.stdout:1074: writing smithi07929762-9 from 1956409 to 2512586 tid 2 2024-04-24T01:59:31.064 INFO:tasks.rados.rados.0.smithi079.stdout:1075: write (excl) oid 35 current snap is 120 2024-04-24T01:59:31.064 INFO:tasks.rados.rados.0.smithi079.stdout:1066: expect (ObjNum 437 snap 99 seq_num 437) 2024-04-24T01:59:31.190 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T01:59:31.205 INFO:tasks.rados.rados.0.smithi079.stdout:1068: expect (ObjNum 276 snap 71 seq_num 276) 2024-04-24T01:59:31.296 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:59:31.294+0000 7f12512ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:59:31.300 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:59:31.299+0000 7f12512ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:59:31.430 INFO:tasks.rados.rados.0.smithi079.stdout:1070: finishing write tid 1 to smithi07929762-23 2024-04-24T01:59:31.430 INFO:tasks.rados.rados.0.smithi079.stdout:1069: finishing write tid 1 to smithi07929762-30 2024-04-24T01:59:31.430 INFO:tasks.rados.rados.0.smithi079.stdout:1067: finishing write tid 1 to smithi07929762-16 2024-04-24T01:59:31.430 INFO:tasks.rados.rados.0.smithi079.stdout:1070: finishing write tid 2 to smithi07929762-23 2024-04-24T01:59:31.430 INFO:tasks.rados.rados.0.smithi079.stdout:1070: finishing write tid 3 to smithi07929762-23 2024-04-24T01:59:31.430 INFO:tasks.rados.rados.0.smithi079.stdout:1075: seq_num 532 ranges {541324=460825,1498538=416188,2393385=547939} 2024-04-24T01:59:31.435 INFO:tasks.rados.rados.0.smithi079.stdout:1075: writing smithi07929762-35 from 541324 to 1002149 tid 1 2024-04-24T01:59:31.440 INFO:tasks.rados.rados.0.smithi079.stdout:1075: writing smithi07929762-35 from 1498538 to 1914726 tid 2 2024-04-24T01:59:31.448 INFO:tasks.rados.rados.0.smithi079.stdout:1075: writing smithi07929762-35 from 2393385 to 2941324 tid 3 2024-04-24T01:59:31.448 INFO:tasks.rados.rados.0.smithi079.stdout:1066: done (8 left) 2024-04-24T01:59:31.448 INFO:tasks.rados.rados.0.smithi079.stdout:1068: done (7 left) 2024-04-24T01:59:31.448 INFO:tasks.rados.rados.0.smithi079.stdout:1076: read oid 48 snap -1 2024-04-24T01:59:31.448 INFO:tasks.rados.rados.0.smithi079.stdout:1076: expect (ObjNum 511 snap 115 seq_num 511) 2024-04-24T01:59:31.449 INFO:tasks.rados.rados.0.smithi079.stdout:1070: finishing write tid 4 to smithi07929762-23 2024-04-24T01:59:31.449 INFO:tasks.rados.rados.0.smithi079.stdout:1070: finishing write tid 5 to smithi07929762-23 2024-04-24T01:59:31.449 INFO:tasks.rados.rados.0.smithi079.stdout:1070: oid 23 updating version 0 to 86 2024-04-24T01:59:31.449 INFO:tasks.rados.rados.0.smithi079.stdout:1070: oid 23 version 86 is already newer than 86 2024-04-24T01:59:31.449 INFO:tasks.rados.rados.0.smithi079.stdout:1070: oid 23 version 86 is already newer than 86 2024-04-24T01:59:31.449 INFO:tasks.rados.rados.0.smithi079.stdout:1070: oid 23 version 86 is already newer than 86 2024-04-24T01:59:31.449 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 23 v 86 (ObjNum 528 snap 120 seq_num 528) dirty exists 2024-04-24T01:59:31.449 INFO:tasks.rados.rados.0.smithi079.stdout:1070: left oid 23 (ObjNum 528 snap 120 seq_num 528) 2024-04-24T01:59:31.449 INFO:tasks.rados.rados.0.smithi079.stdout:1069: finishing write tid 2 to smithi07929762-30 2024-04-24T01:59:31.449 INFO:tasks.rados.rados.0.smithi079.stdout:1077: read oid 37 snap -1 2024-04-24T01:59:31.449 INFO:tasks.rados.rados.0.smithi079.stdout:1077: expect (ObjNum 525 snap 120 seq_num 525) 2024-04-24T01:59:31.449 INFO:tasks.rados.rados.0.smithi079.stdout:1070: done (8 left) 2024-04-24T01:59:31.449 INFO:tasks.rados.rados.0.smithi079.stdout:1078: delete oid 27 current snap is 120 2024-04-24T01:59:31.449 INFO:tasks.rados.rados.0.smithi079.stdout:1067: finishing write tid 2 to smithi07929762-16 2024-04-24T01:59:31.449 INFO:tasks.rados.rados.0.smithi079.stdout:1069: finishing write tid 3 to smithi07929762-30 2024-04-24T01:59:31.449 INFO:tasks.rados.rados.0.smithi079.stdout:1067: finishing write tid 3 to smithi07929762-16 2024-04-24T01:59:31.450 INFO:tasks.rados.rados.0.smithi079.stdout:1069: finishing write tid 4 to smithi07929762-30 2024-04-24T01:59:31.450 INFO:tasks.rados.rados.0.smithi079.stdout:1067: finishing write tid 4 to smithi07929762-16 2024-04-24T01:59:31.450 INFO:tasks.rados.rados.0.smithi079.stdout:1069: finishing write tid 5 to smithi07929762-30 2024-04-24T01:59:31.450 INFO:tasks.rados.rados.0.smithi079.stdout:1069: oid 30 updating version 0 to 110 2024-04-24T01:59:31.450 INFO:tasks.rados.rados.0.smithi079.stdout:1069: oid 30 updating version 110 to 113 2024-04-24T01:59:31.450 INFO:tasks.rados.rados.0.smithi079.stdout:1069: oid 30 version 113 is already newer than 113 2024-04-24T01:59:31.450 INFO:tasks.rados.rados.0.smithi079.stdout:1069: oid 30 version 113 is already newer than 113 2024-04-24T01:59:31.450 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 30 v 113 (ObjNum 527 snap 120 seq_num 527) dirty exists 2024-04-24T01:59:31.450 INFO:tasks.rados.rados.0.smithi079.stdout:1069: left oid 30 (ObjNum 527 snap 120 seq_num 527) 2024-04-24T01:59:31.450 INFO:tasks.rados.rados.0.smithi079.stdout:1067: finishing write tid 5 to smithi07929762-16 2024-04-24T01:59:31.450 INFO:tasks.rados.rados.0.smithi079.stdout:1067: oid 16 updating version 0 to 495 2024-04-24T01:59:31.450 INFO:tasks.rados.rados.0.smithi079.stdout:1067: oid 16 version 495 is already newer than 495 2024-04-24T01:59:31.450 INFO:tasks.rados.rados.0.smithi079.stdout:1067: oid 16 version 495 is already newer than 492 2024-04-24T01:59:31.450 INFO:tasks.rados.rados.0.smithi079.stdout:1067: oid 16 version 495 is already newer than 495 2024-04-24T01:59:31.450 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 16 v 495 (ObjNum 526 snap 120 seq_num 526) dirty exists 2024-04-24T01:59:31.450 INFO:tasks.rados.rados.0.smithi079.stdout:1067: left oid 16 (ObjNum 526 snap 120 seq_num 526) 2024-04-24T01:59:31.451 INFO:tasks.rados.rados.0.smithi079.stdout:1071: finishing write tid 1 to smithi07929762-43 2024-04-24T01:59:31.451 INFO:tasks.rados.rados.0.smithi079.stdout:1071: finishing write tid 2 to smithi07929762-43 2024-04-24T01:59:31.451 INFO:tasks.rados.rados.0.smithi079.stdout:1071: finishing write tid 3 to smithi07929762-43 2024-04-24T01:59:31.451 INFO:tasks.rados.rados.0.smithi079.stdout:1071: finishing write tid 4 to smithi07929762-43 2024-04-24T01:59:31.451 INFO:tasks.rados.rados.0.smithi079.stdout:1071: oid 43 updating version 0 to 116 2024-04-24T01:59:31.451 INFO:tasks.rados.rados.0.smithi079.stdout:1071: oid 43 version 116 is already newer than 116 2024-04-24T01:59:31.451 INFO:tasks.rados.rados.0.smithi079.stdout:1071: oid 43 version 116 is already newer than 116 2024-04-24T01:59:31.451 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 43 v 116 (ObjNum 529 snap 120 seq_num 529) dirty exists 2024-04-24T01:59:31.451 INFO:tasks.rados.rados.0.smithi079.stdout:1071: left oid 43 (ObjNum 529 snap 120 seq_num 529) 2024-04-24T01:59:31.451 INFO:tasks.rados.rados.0.smithi079.stdout:1072: finishing write tid 1 to smithi07929762-47 2024-04-24T01:59:31.451 INFO:tasks.rados.rados.0.smithi079.stdout:1072: finishing write tid 2 to smithi07929762-47 2024-04-24T01:59:31.451 INFO:tasks.rados.rados.0.smithi079.stdout:1072: finishing write tid 3 to smithi07929762-47 2024-04-24T01:59:31.451 INFO:tasks.rados.rados.0.smithi079.stdout:1072: finishing write tid 4 to smithi07929762-47 2024-04-24T01:59:31.451 INFO:tasks.rados.rados.0.smithi079.stdout:1072: finishing write tid 5 to smithi07929762-47 2024-04-24T01:59:31.451 INFO:tasks.rados.rados.0.smithi079.stdout:1072: oid 47 updating version 0 to 374 2024-04-24T01:59:31.451 INFO:tasks.rados.rados.0.smithi079.stdout:1072: oid 47 version 374 is already newer than 374 2024-04-24T01:59:31.451 INFO:tasks.rados.rados.0.smithi079.stdout:1072: oid 47 version 374 is already newer than 374 2024-04-24T01:59:31.452 INFO:tasks.rados.rados.0.smithi079.stdout:1072: oid 47 version 374 is already newer than 374 2024-04-24T01:59:31.452 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 47 v 374 (ObjNum 530 snap 120 seq_num 530) dirty exists 2024-04-24T01:59:31.452 INFO:tasks.rados.rados.0.smithi079.stdout:1072: left oid 47 (ObjNum 530 snap 120 seq_num 530) 2024-04-24T01:59:31.452 INFO:tasks.rados.rados.0.smithi079.stdout:1074: finishing write tid 1 to smithi07929762-9 2024-04-24T01:59:31.452 INFO:tasks.rados.rados.0.smithi079.stdout:1074: finishing write tid 2 to smithi07929762-9 2024-04-24T01:59:31.452 INFO:tasks.rados.rados.0.smithi079.stdout:1074: finishing write tid 3 to smithi07929762-9 2024-04-24T01:59:31.452 INFO:tasks.rados.rados.0.smithi079.stdout:1074: finishing write tid 4 to smithi07929762-9 2024-04-24T01:59:31.452 INFO:tasks.rados.rados.0.smithi079.stdout:1074: oid 9 updating version 0 to 321 2024-04-24T01:59:31.452 INFO:tasks.rados.rados.0.smithi079.stdout:1074: oid 9 version 321 is already newer than 321 2024-04-24T01:59:31.452 INFO:tasks.rados.rados.0.smithi079.stdout:1074: oid 9 version 321 is already newer than 321 2024-04-24T01:59:31.452 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 9 v 321 (ObjNum 531 snap 120 seq_num 531) dirty exists 2024-04-24T01:59:31.452 INFO:tasks.rados.rados.0.smithi079.stdout:1074: left oid 9 (ObjNum 531 snap 120 seq_num 531) 2024-04-24T01:59:31.488 INFO:tasks.rados.rados.0.smithi079.stdout:1075: finishing write tid 1 to smithi07929762-35 2024-04-24T01:59:31.509 INFO:tasks.rados.rados.0.smithi079.stdout:1076: expect (ObjNum 511 snap 115 seq_num 511) 2024-04-24T01:59:31.678 INFO:tasks.rados.rados.0.smithi079.stdout:1075: finishing write tid 2 to smithi07929762-35 2024-04-24T01:59:31.679 INFO:tasks.rados.rados.0.smithi079.stdout:1077: expect (ObjNum 525 snap 120 seq_num 525) 2024-04-24T01:59:31.820 INFO:tasks.rados.rados.0.smithi079.stdout:1075: finishing write tid 3 to smithi07929762-35 2024-04-24T01:59:31.820 INFO:tasks.rados.rados.0.smithi079.stdout:1075: finishing write tid 4 to smithi07929762-35 2024-04-24T01:59:31.820 INFO:tasks.rados.rados.0.smithi079.stdout:1075: finishing write tid 5 to smithi07929762-35 2024-04-24T01:59:31.820 INFO:tasks.rados.rados.0.smithi079.stdout:1075: oid 35 updating version 0 to 310 2024-04-24T01:59:31.820 INFO:tasks.rados.rados.0.smithi079.stdout:1075: oid 35 version 310 is already newer than 307 2024-04-24T01:59:31.820 INFO:tasks.rados.rados.0.smithi079.stdout:1075: oid 35 version 310 is already newer than 310 2024-04-24T01:59:31.820 INFO:tasks.rados.rados.0.smithi079.stdout:1075: oid 35 version 310 is already newer than 310 2024-04-24T01:59:31.820 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 35 v 310 (ObjNum 532 snap 120 seq_num 532) dirty exists 2024-04-24T01:59:31.820 INFO:tasks.rados.rados.0.smithi079.stdout:1075: left oid 35 (ObjNum 532 snap 120 seq_num 532) 2024-04-24T01:59:31.820 INFO:tasks.rados.rados.0.smithi079.stdout:1067: done (8 left) 2024-04-24T01:59:31.820 INFO:tasks.rados.rados.0.smithi079.stdout:1069: done (7 left) 2024-04-24T01:59:31.820 INFO:tasks.rados.rados.0.smithi079.stdout:1071: done (6 left) 2024-04-24T01:59:31.821 INFO:tasks.rados.rados.0.smithi079.stdout:1072: done (5 left) 2024-04-24T01:59:31.821 INFO:tasks.rados.rados.0.smithi079.stdout:1074: done (4 left) 2024-04-24T01:59:31.821 INFO:tasks.rados.rados.0.smithi079.stdout:1075: done (3 left) 2024-04-24T01:59:31.821 INFO:tasks.rados.rados.0.smithi079.stdout:1076: done (2 left) 2024-04-24T01:59:31.821 INFO:tasks.rados.rados.0.smithi079.stdout:1077: done (1 left) 2024-04-24T01:59:31.821 INFO:tasks.rados.rados.0.smithi079.stdout:1078: done (0 left) 2024-04-24T01:59:31.821 INFO:tasks.rados.rados.0.smithi079.stdout:1079: snap_create 2024-04-24T01:59:32.105 INFO:tasks.rados.rados.0.smithi079.stdout:1079: done (0 left) 2024-04-24T01:59:32.105 INFO:tasks.rados.rados.0.smithi079.stdout:1080: delete oid 17 current snap is 121 2024-04-24T01:59:32.157 INFO:tasks.rados.rados.0.smithi079.stdout:1080: done (0 left) 2024-04-24T01:59:32.157 INFO:tasks.rados.rados.0.smithi079.stdout:1081: write oid 1 current snap is 121 2024-04-24T01:59:32.157 INFO:tasks.rados.rados.0.smithi079.stdout:1081: seq_num 535 ranges {591745=402925,1790337=563515} 2024-04-24T01:59:32.162 INFO:tasks.rados.rados.0.smithi079.stdout:1081: writing smithi07929762-1 from 591745 to 994670 tid 1 2024-04-24T01:59:32.169 INFO:tasks.rados.rados.0.smithi079.stdout:1081: writing smithi07929762-1 from 1790337 to 2353852 tid 2 2024-04-24T01:59:32.169 INFO:tasks.rados.rados.0.smithi079.stdout:1082: snap_remove snap 120 2024-04-24T01:59:33.108 INFO:tasks.rados.rados.0.smithi079.stdout:1082: done (1 left) 2024-04-24T01:59:33.108 INFO:tasks.rados.rados.0.smithi079.stdout:1083: read oid 7 snap 115 2024-04-24T01:59:33.108 INFO:tasks.rados.rados.0.smithi079.stdout:1083: expect (ObjNum 375 snap 90 seq_num 375) 2024-04-24T01:59:33.108 INFO:tasks.rados.rados.0.smithi079.stdout:1084: write (excl) oid 9 current snap is 121 2024-04-24T01:59:33.108 INFO:tasks.rados.rados.0.smithi079.stdout:1081: finishing write tid 1 to smithi07929762-1 2024-04-24T01:59:33.108 INFO:tasks.rados.rados.0.smithi079.stdout:1084: seq_num 536 ranges {608552=478174,1848819=707173} 2024-04-24T01:59:33.113 INFO:tasks.rados.rados.0.smithi079.stdout:1084: writing smithi07929762-9 from 608552 to 1086726 tid 1 2024-04-24T01:59:33.123 INFO:tasks.rados.rados.0.smithi079.stdout:1084: writing smithi07929762-9 from 1848819 to 2555992 tid 2 2024-04-24T01:59:33.123 INFO:tasks.rados.rados.0.smithi079.stdout:1085: read oid 47 snap -1 2024-04-24T01:59:33.124 INFO:tasks.rados.rados.0.smithi079.stdout:1085: expect (ObjNum 530 snap 120 seq_num 530) 2024-04-24T01:59:33.124 INFO:tasks.rados.rados.0.smithi079.stdout:1086: read oid 13 snap -1 2024-04-24T01:59:33.124 INFO:tasks.rados.rados.0.smithi079.stdout:1086: expect deleted 2024-04-24T01:59:33.124 INFO:tasks.rados.rados.0.smithi079.stdout:1087: read oid 20 snap -1 2024-04-24T01:59:33.124 INFO:tasks.rados.rados.0.smithi079.stdout:1087: expect (ObjNum 486 snap 111 seq_num 486) 2024-04-24T01:59:33.124 INFO:tasks.rados.rados.0.smithi079.stdout:1088: read oid 14 snap -1 2024-04-24T01:59:33.124 INFO:tasks.rados.rados.0.smithi079.stdout:1088: expect (ObjNum 506 snap 112 seq_num 506) 2024-04-24T01:59:33.124 INFO:tasks.rados.rados.0.smithi079.stdout:1089: write oid 8 current snap is 121 2024-04-24T01:59:33.124 INFO:tasks.rados.rados.0.smithi079.stdout:1081: finishing write tid 2 to smithi07929762-1 2024-04-24T01:59:33.124 INFO:tasks.rados.rados.0.smithi079.stdout:1089: seq_num 537 ranges {625359=553423,1790948=567184,2930500=94859} 2024-04-24T01:59:33.131 INFO:tasks.rados.rados.0.smithi079.stdout:1089: writing smithi07929762-8 from 625359 to 1178782 tid 1 2024-04-24T01:59:33.138 INFO:tasks.rados.rados.0.smithi079.stdout:1089: writing smithi07929762-8 from 1790948 to 2358132 tid 2 2024-04-24T01:59:33.142 INFO:tasks.rados.rados.0.smithi079.stdout:1089: writing smithi07929762-8 from 2930500 to 3025359 tid 3 2024-04-24T01:59:33.142 INFO:tasks.rados.rados.0.smithi079.stdout:1090: read oid 32 snap -1 2024-04-24T01:59:33.142 INFO:tasks.rados.rados.0.smithi079.stdout:1090: expect (ObjNum 374 snap 90 seq_num 374) 2024-04-24T01:59:33.142 INFO:tasks.rados.rados.0.smithi079.stdout:1091: read oid 17 snap -1 2024-04-24T01:59:33.142 INFO:tasks.rados.rados.0.smithi079.stdout:1091: expect deleted 2024-04-24T01:59:33.142 INFO:tasks.rados.rados.0.smithi079.stdout:1092: snap_remove snap 114 2024-04-24T01:59:33.394 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T01:59:33.498 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:59:33.496+0000 7fadc520f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:59:33.501 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:59:33.501+0000 7fadc520f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:59:34.109 INFO:tasks.rados.rados.0.smithi079.stdout:1092: done (10 left) 2024-04-24T01:59:34.109 INFO:tasks.rados.rados.0.smithi079.stdout:1093: read oid 19 snap -1 2024-04-24T01:59:34.109 INFO:tasks.rados.rados.0.smithi079.stdout:1093: expect deleted 2024-04-24T01:59:34.109 INFO:tasks.rados.rados.0.smithi079.stdout:1094: write (excl) oid 35 current snap is 121 2024-04-24T01:59:34.109 INFO:tasks.rados.rados.0.smithi079.stdout:1081: finishing write tid 3 to smithi07929762-1 2024-04-24T01:59:34.109 INFO:tasks.rados.rados.0.smithi079.stdout:1094: seq_num 538 ranges {642166=628672,1849430=710842} 2024-04-24T01:59:34.116 INFO:tasks.rados.rados.0.smithi079.stdout:1094: writing smithi07929762-35 from 642166 to 1270838 tid 1 2024-04-24T01:59:34.124 INFO:tasks.rados.rados.0.smithi079.stdout:1094: writing smithi07929762-35 from 1849430 to 2560272 tid 2 2024-04-24T01:59:34.124 INFO:tasks.rados.rados.0.smithi079.stdout:1095: delete oid 23 current snap is 121 2024-04-24T01:59:34.124 INFO:tasks.rados.rados.0.smithi079.stdout:1081: finishing write tid 4 to smithi07929762-1 2024-04-24T01:59:34.124 INFO:tasks.rados.rados.0.smithi079.stdout:1081: oid 1 updating version 0 to 502 2024-04-24T01:59:34.124 INFO:tasks.rados.rados.0.smithi079.stdout:1081: oid 1 version 502 is already newer than 502 2024-04-24T01:59:34.124 INFO:tasks.rados.rados.0.smithi079.stdout:1081: oid 1 version 502 is already newer than 502 2024-04-24T01:59:34.124 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 1 v 502 (ObjNum 535 snap 121 seq_num 535) dirty exists 2024-04-24T01:59:34.124 INFO:tasks.rados.rados.0.smithi079.stdout:1081: left oid 1 (ObjNum 535 snap 121 seq_num 535) 2024-04-24T01:59:34.124 INFO:tasks.rados.rados.0.smithi079.stdout:1083: expect (ObjNum 375 snap 90 seq_num 375) 2024-04-24T01:59:34.235 INFO:tasks.rados.rados.0.smithi079.stdout:1090: expect (ObjNum 374 snap 90 seq_num 374) 2024-04-24T01:59:34.367 INFO:tasks.rados.rados.0.smithi079.stdout:1084: finishing write tid 1 to smithi07929762-9 2024-04-24T01:59:34.367 INFO:tasks.rados.rados.0.smithi079.stdout:1081: done (12 left) 2024-04-24T01:59:34.367 INFO:tasks.rados.rados.0.smithi079.stdout:1083: done (11 left) 2024-04-24T01:59:34.368 INFO:tasks.rados.rados.0.smithi079.stdout:1090: done (10 left) 2024-04-24T01:59:34.368 INFO:tasks.rados.rados.0.smithi079.stdout:1091: done (9 left) 2024-04-24T01:59:34.368 INFO:tasks.rados.rados.0.smithi079.stdout:1095: done (8 left) 2024-04-24T01:59:34.368 INFO:tasks.rados.rados.0.smithi079.stdout:1096: write oid 18 current snap is 121 2024-04-24T01:59:34.368 INFO:tasks.rados.rados.0.smithi079.stdout:1088: expect (ObjNum 506 snap 112 seq_num 506) 2024-04-24T01:59:34.462 INFO:tasks.rados.rados.0.smithi079.stdout:1089: finishing write tid 1 to smithi07929762-8 2024-04-24T01:59:34.462 INFO:tasks.rados.rados.0.smithi079.stdout:1085: expect (ObjNum 530 snap 120 seq_num 530) 2024-04-24T01:59:34.612 INFO:tasks.rados.rados.0.smithi079.stdout:1089: finishing write tid 2 to smithi07929762-8 2024-04-24T01:59:34.612 INFO:tasks.rados.rados.0.smithi079.stdout:1084: finishing write tid 2 to smithi07929762-9 2024-04-24T01:59:34.612 INFO:tasks.rados.rados.0.smithi079.stdout:1084: finishing write tid 3 to smithi07929762-9 2024-04-24T01:59:34.612 INFO:tasks.rados.rados.0.smithi079.stdout:1089: finishing write tid 3 to smithi07929762-8 2024-04-24T01:59:34.612 INFO:tasks.rados.rados.0.smithi079.stdout:1084: finishing write tid 4 to smithi07929762-9 2024-04-24T01:59:34.612 INFO:tasks.rados.rados.0.smithi079.stdout:1084: oid 9 updating version 0 to 322 2024-04-24T01:59:34.612 INFO:tasks.rados.rados.0.smithi079.stdout:1084: oid 9 updating version 322 to 324 2024-04-24T01:59:34.612 INFO:tasks.rados.rados.0.smithi079.stdout:1084: oid 9 version 324 is already newer than 324 2024-04-24T01:59:34.613 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 9 v 324 (ObjNum 536 snap 121 seq_num 536) dirty exists 2024-04-24T01:59:34.613 INFO:tasks.rados.rados.0.smithi079.stdout:1084: left oid 9 (ObjNum 536 snap 121 seq_num 536) 2024-04-24T01:59:34.613 INFO:tasks.rados.rados.0.smithi079.stdout:1089: finishing write tid 4 to smithi07929762-8 2024-04-24T01:59:34.613 INFO:tasks.rados.rados.0.smithi079.stdout:1096: seq_num 540 ranges {675780=495523,1682747=714511,3012769=63011} 2024-04-24T01:59:34.618 INFO:tasks.rados.rados.0.smithi079.stdout:1096: writing smithi07929762-18 from 675780 to 1171303 tid 1 2024-04-24T01:59:34.625 INFO:tasks.rados.rados.0.smithi079.stdout:1096: writing smithi07929762-18 from 1682747 to 2397258 tid 2 2024-04-24T01:59:34.630 INFO:tasks.rados.rados.0.smithi079.stdout:1096: writing smithi07929762-18 from 3012769 to 3075780 tid 3 2024-04-24T01:59:34.630 INFO:tasks.rados.rados.0.smithi079.stdout:1084: done (8 left) 2024-04-24T01:59:34.630 INFO:tasks.rados.rados.0.smithi079.stdout:1085: done (7 left) 2024-04-24T01:59:34.630 INFO:tasks.rados.rados.0.smithi079.stdout:1086: done (6 left) 2024-04-24T01:59:34.630 INFO:tasks.rados.rados.0.smithi079.stdout:1088: done (5 left) 2024-04-24T01:59:34.630 INFO:tasks.rados.rados.0.smithi079.stdout:1097: write oid 7 current snap is 121 2024-04-24T01:59:34.630 INFO:tasks.rados.rados.0.smithi079.stdout:1089: finishing write tid 5 to smithi07929762-8 2024-04-24T01:59:34.630 INFO:tasks.rados.rados.0.smithi079.stdout:1089: oid 8 updating version 0 to 30 2024-04-24T01:59:34.630 INFO:tasks.rados.rados.0.smithi079.stdout:1089: oid 8 version 30 is already newer than 30 2024-04-24T01:59:34.630 INFO:tasks.rados.rados.0.smithi079.stdout:1089: oid 8 version 30 is already newer than 30 2024-04-24T01:59:34.630 INFO:tasks.rados.rados.0.smithi079.stdout:1089: oid 8 version 30 is already newer than 30 2024-04-24T01:59:34.630 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 8 v 30 (ObjNum 537 snap 121 seq_num 537) dirty exists 2024-04-24T01:59:34.630 INFO:tasks.rados.rados.0.smithi079.stdout:1089: left oid 8 (ObjNum 537 snap 121 seq_num 537) 2024-04-24T01:59:34.630 INFO:tasks.rados.rados.0.smithi079.stdout:1097: seq_num 541 ranges {692587=570772,2024876=574522,3040192=52395} 2024-04-24T01:59:34.636 INFO:tasks.rados.rados.0.smithi079.stdout:1097: writing smithi07929762-7 from 692587 to 1263359 tid 1 2024-04-24T01:59:34.644 INFO:tasks.rados.rados.0.smithi079.stdout:1097: writing smithi07929762-7 from 2024876 to 2599398 tid 2 2024-04-24T01:59:34.647 INFO:tasks.rados.rados.0.smithi079.stdout:1097: writing smithi07929762-7 from 3040192 to 3092587 tid 3 2024-04-24T01:59:34.647 INFO:tasks.rados.rados.0.smithi079.stdout:1089: done (5 left) 2024-04-24T01:59:34.647 INFO:tasks.rados.rados.0.smithi079.stdout:1098: write (excl) oid 38 current snap is 121 2024-04-24T01:59:34.647 INFO:tasks.rados.rados.0.smithi079.stdout:1087: expect (ObjNum 486 snap 111 seq_num 486) 2024-04-24T01:59:34.763 INFO:tasks.rados.rados.0.smithi079.stdout:1094: finishing write tid 1 to smithi07929762-35 2024-04-24T01:59:34.764 INFO:tasks.rados.rados.0.smithi079.stdout:1094: finishing write tid 2 to smithi07929762-35 2024-04-24T01:59:34.764 INFO:tasks.rados.rados.0.smithi079.stdout:1094: finishing write tid 3 to smithi07929762-35 2024-04-24T01:59:34.764 INFO:tasks.rados.rados.0.smithi079.stdout:1094: finishing write tid 4 to smithi07929762-35 2024-04-24T01:59:34.764 INFO:tasks.rados.rados.0.smithi079.stdout:1094: oid 35 updating version 0 to 313 2024-04-24T01:59:34.764 INFO:tasks.rados.rados.0.smithi079.stdout:1094: oid 35 version 313 is already newer than 313 2024-04-24T01:59:34.764 INFO:tasks.rados.rados.0.smithi079.stdout:1094: oid 35 version 313 is already newer than 311 2024-04-24T01:59:34.764 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 35 v 313 (ObjNum 538 snap 121 seq_num 538) dirty exists 2024-04-24T01:59:34.764 INFO:tasks.rados.rados.0.smithi079.stdout:1094: left oid 35 (ObjNum 538 snap 121 seq_num 538) 2024-04-24T01:59:34.764 INFO:tasks.rados.rados.0.smithi079.stdout:1096: finishing write tid 1 to smithi07929762-18 2024-04-24T01:59:34.764 INFO:tasks.rados.rados.0.smithi079.stdout:1098: seq_num 542 ranges {709394=646021,2083358=718180} 2024-04-24T01:59:34.770 INFO:tasks.rados.rados.0.smithi079.stdout:1098: writing smithi07929762-38 from 709394 to 1355415 tid 1 2024-04-24T01:59:34.779 INFO:tasks.rados.rados.0.smithi079.stdout:1098: writing smithi07929762-38 from 2083358 to 2801538 tid 2 2024-04-24T01:59:34.780 INFO:tasks.rados.rados.0.smithi079.stdout:1087: done (5 left) 2024-04-24T01:59:34.780 INFO:tasks.rados.rados.0.smithi079.stdout:1093: done (4 left) 2024-04-24T01:59:34.780 INFO:tasks.rados.rados.0.smithi079.stdout:1094: done (3 left) 2024-04-24T01:59:34.780 INFO:tasks.rados.rados.0.smithi079.stdout:1099: snap_create 2024-04-24T01:59:34.780 INFO:tasks.rados.rados.0.smithi079.stdout:1097: finishing write tid 1 to smithi07929762-7 2024-04-24T01:59:34.780 INFO:tasks.rados.rados.0.smithi079.stdout:1096: finishing write tid 2 to smithi07929762-18 2024-04-24T01:59:34.780 INFO:tasks.rados.rados.0.smithi079.stdout:1097: finishing write tid 2 to smithi07929762-7 2024-04-24T01:59:34.780 INFO:tasks.rados.rados.0.smithi079.stdout:1096: finishing write tid 3 to smithi07929762-18 2024-04-24T01:59:34.780 INFO:tasks.rados.rados.0.smithi079.stdout:1097: finishing write tid 3 to smithi07929762-7 2024-04-24T01:59:34.780 INFO:tasks.rados.rados.0.smithi079.stdout:1097: finishing write tid 4 to smithi07929762-7 2024-04-24T01:59:34.780 INFO:tasks.rados.rados.0.smithi079.stdout:1097: finishing write tid 5 to smithi07929762-7 2024-04-24T01:59:34.780 INFO:tasks.rados.rados.0.smithi079.stdout:1097: oid 7 updating version 0 to 247 2024-04-24T01:59:34.780 INFO:tasks.rados.rados.0.smithi079.stdout:1097: oid 7 version 247 is already newer than 247 2024-04-24T01:59:34.780 INFO:tasks.rados.rados.0.smithi079.stdout:1097: oid 7 version 247 is already newer than 247 2024-04-24T01:59:34.780 INFO:tasks.rados.rados.0.smithi079.stdout:1097: oid 7 version 247 is already newer than 247 2024-04-24T01:59:34.780 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 7 v 247 (ObjNum 541 snap 121 seq_num 541) dirty exists 2024-04-24T01:59:34.781 INFO:tasks.rados.rados.0.smithi079.stdout:1097: left oid 7 (ObjNum 541 snap 121 seq_num 541) 2024-04-24T01:59:34.781 INFO:tasks.rados.rados.0.smithi079.stdout:1096: finishing write tid 4 to smithi07929762-18 2024-04-24T01:59:34.781 INFO:tasks.rados.rados.0.smithi079.stdout:1096: finishing write tid 5 to smithi07929762-18 2024-04-24T01:59:34.781 INFO:tasks.rados.rados.0.smithi079.stdout:1096: oid 18 updating version 0 to 509 2024-04-24T01:59:34.781 INFO:tasks.rados.rados.0.smithi079.stdout:1096: oid 18 version 509 is already newer than 509 2024-04-24T01:59:34.781 INFO:tasks.rados.rados.0.smithi079.stdout:1096: oid 18 version 509 is already newer than 509 2024-04-24T01:59:34.781 INFO:tasks.rados.rados.0.smithi079.stdout:1096: oid 18 version 509 is already newer than 509 2024-04-24T01:59:34.781 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 18 v 509 (ObjNum 540 snap 121 seq_num 540) dirty exists 2024-04-24T01:59:34.781 INFO:tasks.rados.rados.0.smithi079.stdout:1096: left oid 18 (ObjNum 540 snap 121 seq_num 540) 2024-04-24T01:59:34.803 INFO:tasks.rados.rados.0.smithi079.stdout:1098: finishing write tid 1 to smithi07929762-38 2024-04-24T01:59:34.820 INFO:tasks.rados.rados.0.smithi079.stdout:1098: finishing write tid 2 to smithi07929762-38 2024-04-24T01:59:34.825 INFO:tasks.rados.rados.0.smithi079.stdout:1098: finishing write tid 3 to smithi07929762-38 2024-04-24T01:59:34.826 INFO:tasks.rados.rados.0.smithi079.stdout:1098: finishing write tid 4 to smithi07929762-38 2024-04-24T01:59:34.826 INFO:tasks.rados.rados.0.smithi079.stdout:1098: oid 38 updating version 0 to 252 2024-04-24T01:59:34.826 INFO:tasks.rados.rados.0.smithi079.stdout:1098: oid 38 version 252 is already newer than 250 2024-04-24T01:59:34.826 INFO:tasks.rados.rados.0.smithi079.stdout:1098: oid 38 version 252 is already newer than 252 2024-04-24T01:59:34.826 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 38 v 252 (ObjNum 542 snap 121 seq_num 542) dirty exists 2024-04-24T01:59:34.826 INFO:tasks.rados.rados.0.smithi079.stdout:1098: left oid 38 (ObjNum 542 snap 121 seq_num 542) 2024-04-24T01:59:35.121 INFO:tasks.rados.rados.0.smithi079.stdout:1096: done (3 left) 2024-04-24T01:59:35.121 INFO:tasks.rados.rados.0.smithi079.stdout:1097: done (2 left) 2024-04-24T01:59:35.121 INFO:tasks.rados.rados.0.smithi079.stdout:1098: done (1 left) 2024-04-24T01:59:35.121 INFO:tasks.rados.rados.0.smithi079.stdout:1099: done (0 left) 2024-04-24T01:59:35.121 INFO:tasks.rados.rados.0.smithi079.stdout:1100: write oid 2 current snap is 122 2024-04-24T01:59:35.121 INFO:tasks.rados.rados.0.smithi079.stdout:1100: seq_num 543 ranges {726201=721270,2141840=578191} 2024-04-24T01:59:35.128 INFO:tasks.rados.rados.0.smithi079.stdout:1100: writing smithi07929762-2 from 726201 to 1447471 tid 1 2024-04-24T01:59:35.135 INFO:tasks.rados.rados.0.smithi079.stdout:1100: writing smithi07929762-2 from 2141840 to 2720031 tid 2 2024-04-24T01:59:35.136 INFO:tasks.rados.rados.0.smithi079.stdout:1101: read oid 6 snap -1 2024-04-24T01:59:35.136 INFO:tasks.rados.rados.0.smithi079.stdout:1101: expect (ObjNum 276 snap 71 seq_num 276) 2024-04-24T01:59:35.136 INFO:tasks.rados.rados.0.smithi079.stdout:1102: delete oid 13 current snap is 122 2024-04-24T01:59:35.170 INFO:tasks.rados.rados.0.smithi079.stdout:1102: done (2 left) 2024-04-24T01:59:35.170 INFO:tasks.rados.rados.0.smithi079.stdout:1103: snap_remove snap 118 2024-04-24T01:59:35.344 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-04-24T01:59:35.345 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-04-24T01:59:35.345 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-04-24T01:59:35.346 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-04-24T01:59:35.453 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:59:35.448+0000 7f999846d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:59:35.457 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:59:35.456+0000 7f999846d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:59:35.471 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T01:59:35.471 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T01:59:35.471 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T01:59:35.471 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T01:59:35.471 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T01:59:35.471 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T01:59:35.471 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T01:59:35.471 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T01:59:35.471 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T01:59:35.471 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T01:59:35.471 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T01:59:35.472 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T01:59:35.474 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T01:59:35.474 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T01:59:35.475 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-04-24T01:59:35.582 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:59:35.576+0000 7f942dd54640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:59:35.585 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:59:35.585+0000 7f942dd54640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:59:35.596 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T01:59:35.600 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T01:59:35.600 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T01:59:35.600 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T01:59:35.600 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T01:59:35.600 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T01:59:35.600 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T01:59:35.600 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T01:59:35.600 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T01:59:35.600 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T01:59:35.600 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T01:59:35.600 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T01:59:35.601 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T01:59:35.603 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T01:59:35.603 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T01:59:35.603 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-04-24T01:59:35.703 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:59:35.700+0000 7feee286d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:59:35.706 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:59:35.706+0000 7feee286d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:59:35.761 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:59:35.755+0000 7f7ea7064640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:59:35.765 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:59:35.764+0000 7f7ea7064640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:59:35.786 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T01:59:35.786 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T01:59:35.786 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T01:59:35.786 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T01:59:35.787 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T01:59:35.787 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T01:59:35.787 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T01:59:35.787 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T01:59:35.787 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T01:59:35.787 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T01:59:35.787 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T01:59:35.787 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T01:59:35.789 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T01:59:35.789 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T01:59:35.789 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-04-24T01:59:35.897 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:59:35.891+0000 7f656c46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:59:35.901 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:59:35.900+0000 7f656c46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:59:35.916 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T01:59:35.917 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T01:59:35.917 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T01:59:35.917 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T01:59:35.917 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T01:59:35.917 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T01:59:35.917 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T01:59:35.917 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T01:59:35.917 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T01:59:35.917 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T01:59:35.917 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T01:59:35.917 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T01:59:35.920 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T01:59:35.920 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T01:59:36.123 INFO:tasks.rados.rados.0.smithi079.stdout:1103: done (2 left) 2024-04-24T01:59:36.123 INFO:tasks.rados.rados.0.smithi079.stdout:1104: snap_remove snap 115 2024-04-24T01:59:37.128 INFO:tasks.rados.rados.0.smithi079.stdout:1104: done (2 left) 2024-04-24T01:59:37.128 INFO:tasks.rados.rados.0.smithi079.stdout:1105: snap_create 2024-04-24T01:59:37.128 INFO:tasks.rados.rados.0.smithi079.stdout:1101: expect (ObjNum 276 snap 71 seq_num 276) 2024-04-24T01:59:37.364 INFO:tasks.rados.rados.0.smithi079.stdout:1100: finishing write tid 1 to smithi07929762-2 2024-04-24T01:59:37.364 INFO:tasks.rados.rados.0.smithi079.stdout:1100: finishing write tid 2 to smithi07929762-2 2024-04-24T01:59:37.364 INFO:tasks.rados.rados.0.smithi079.stdout:1100: finishing write tid 3 to smithi07929762-2 2024-04-24T01:59:37.364 INFO:tasks.rados.rados.0.smithi079.stdout:1100: finishing write tid 4 to smithi07929762-2 2024-04-24T01:59:37.364 INFO:tasks.rados.rados.0.smithi079.stdout:1100: oid 2 updating version 0 to 62 2024-04-24T01:59:37.364 INFO:tasks.rados.rados.0.smithi079.stdout:1100: oid 2 version 62 is already newer than 62 2024-04-24T01:59:37.364 INFO:tasks.rados.rados.0.smithi079.stdout:1100: oid 2 version 62 is already newer than 62 2024-04-24T01:59:37.364 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 2 v 62 (ObjNum 543 snap 122 seq_num 543) dirty exists 2024-04-24T01:59:37.364 INFO:tasks.rados.rados.0.smithi079.stdout:1100: left oid 2 (ObjNum 543 snap 122 seq_num 543) 2024-04-24T01:59:37.809 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T01:59:37.914 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:59:37.912+0000 7f429b8ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:59:37.918 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:59:37.917+0000 7f429b8ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:59:38.128 INFO:tasks.rados.rados.0.smithi079.stdout:1100: done (2 left) 2024-04-24T01:59:38.128 INFO:tasks.rados.rados.0.smithi079.stdout:1101: done (1 left) 2024-04-24T01:59:38.128 INFO:tasks.rados.rados.0.smithi079.stdout:1105: done (0 left) 2024-04-24T01:59:38.128 INFO:tasks.rados.rados.0.smithi079.stdout:1106: write (excl) oid 40 current snap is 123 2024-04-24T01:59:38.128 INFO:tasks.rados.rados.0.smithi079.stdout:1106: seq_num 545 ranges {759815=471768,1742451=465507,2917178=242637} 2024-04-24T01:59:38.136 INFO:tasks.rados.rados.0.smithi079.stdout:1106: writing smithi07929762-40 from 759815 to 1231583 tid 1 2024-04-24T01:59:38.147 INFO:tasks.rados.rados.0.smithi079.stdout:1106: writing smithi07929762-40 from 1742451 to 2207958 tid 2 2024-04-24T01:59:38.153 INFO:tasks.rados.rados.0.smithi079.stdout:1106: writing smithi07929762-40 from 2917178 to 3159815 tid 3 2024-04-24T01:59:38.153 INFO:tasks.rados.rados.0.smithi079.stdout:1107: write (excl) oid 48 current snap is 123 2024-04-24T01:59:38.153 INFO:tasks.rados.rados.0.smithi079.stdout:1107: seq_num 546 ranges {776622=547017,1800933=725518,2944601=232021} 2024-04-24T01:59:38.162 INFO:tasks.rados.rados.0.smithi079.stdout:1107: writing smithi07929762-48 from 776622 to 1323639 tid 1 2024-04-24T01:59:38.170 INFO:tasks.rados.rados.0.smithi079.stdout:1107: writing smithi07929762-48 from 1800933 to 2526451 tid 2 2024-04-24T01:59:38.174 INFO:tasks.rados.rados.0.smithi079.stdout:1107: writing smithi07929762-48 from 2944601 to 3176622 tid 3 2024-04-24T01:59:38.174 INFO:tasks.rados.rados.0.smithi079.stdout:1108: write (excl) oid 13 current snap is 123 2024-04-24T01:59:38.174 INFO:tasks.rados.rados.0.smithi079.stdout:1108: seq_num 547 ranges {793429=622266,1859415=469176,2972024=221405} 2024-04-24T01:59:38.183 INFO:tasks.rados.rados.0.smithi079.stdout:1108: writing smithi07929762-13 from 793429 to 1415695 tid 1 2024-04-24T01:59:38.190 INFO:tasks.rados.rados.0.smithi079.stdout:1108: writing smithi07929762-13 from 1859415 to 2328591 tid 2 2024-04-24T01:59:38.196 INFO:tasks.rados.rados.0.smithi079.stdout:1108: writing smithi07929762-13 from 2972024 to 3193429 tid 3 2024-04-24T01:59:38.196 INFO:tasks.rados.rados.0.smithi079.stdout:1109: write (excl) oid 43 current snap is 123 2024-04-24T01:59:38.196 INFO:tasks.rados.rados.0.smithi079.stdout:1109: seq_num 548 ranges {410236=413868,1234250=729187,2715800=423063} 2024-04-24T01:59:38.202 INFO:tasks.rados.rados.0.smithi079.stdout:1109: writing smithi07929762-43 from 410236 to 824104 tid 1 2024-04-24T01:59:38.214 INFO:tasks.rados.rados.0.smithi079.stdout:1109: writing smithi07929762-43 from 1234250 to 1963437 tid 2 2024-04-24T01:59:38.221 INFO:tasks.rados.rados.0.smithi079.stdout:1109: writing smithi07929762-43 from 2715800 to 3138863 tid 3 2024-04-24T01:59:38.222 INFO:tasks.rados.rados.0.smithi079.stdout:1110: snap_remove snap 113 2024-04-24T01:59:39.149 INFO:tasks.rados.rados.0.smithi079.stdout:1110: done (4 left) 2024-04-24T01:59:39.149 INFO:tasks.rados.rados.0.smithi079.stdout:1111: write oid 3 current snap is 123 2024-04-24T01:59:39.149 INFO:tasks.rados.rados.0.smithi079.stdout:1106: finishing write tid 1 to smithi07929762-40 2024-04-24T01:59:39.149 INFO:tasks.rados.rados.0.smithi079.stdout:1111: seq_num 549 ranges {427043=489117,1576379=472845,2626870=600173} 2024-04-24T01:59:39.154 INFO:tasks.rados.rados.0.smithi079.stdout:1111: writing smithi07929762-3 from 427043 to 916160 tid 1 2024-04-24T01:59:39.160 INFO:tasks.rados.rados.0.smithi079.stdout:1111: writing smithi07929762-3 from 1576379 to 2049224 tid 2 2024-04-24T01:59:39.167 INFO:tasks.rados.rados.0.smithi079.stdout:1111: writing smithi07929762-3 from 2626870 to 3227043 tid 3 2024-04-24T01:59:39.167 INFO:tasks.rados.rados.0.smithi079.stdout:1112: snap_create 2024-04-24T01:59:39.167 INFO:tasks.rados.rados.0.smithi079.stdout:1106: finishing write tid 2 to smithi07929762-40 2024-04-24T01:59:39.167 INFO:tasks.rados.rados.0.smithi079.stdout:1106: finishing write tid 3 to smithi07929762-40 2024-04-24T01:59:39.167 INFO:tasks.rados.rados.0.smithi079.stdout:1106: finishing write tid 4 to smithi07929762-40 2024-04-24T01:59:39.168 INFO:tasks.rados.rados.0.smithi079.stdout:1106: finishing write tid 5 to smithi07929762-40 2024-04-24T01:59:39.168 INFO:tasks.rados.rados.0.smithi079.stdout:1106: oid 40 updating version 0 to 329 2024-04-24T01:59:39.168 INFO:tasks.rados.rados.0.smithi079.stdout:1106: oid 40 updating version 329 to 332 2024-04-24T01:59:39.168 INFO:tasks.rados.rados.0.smithi079.stdout:1106: oid 40 version 332 is already newer than 332 2024-04-24T01:59:39.168 INFO:tasks.rados.rados.0.smithi079.stdout:1106: oid 40 version 332 is already newer than 332 2024-04-24T01:59:39.168 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 40 v 332 (ObjNum 545 snap 123 seq_num 545) dirty exists 2024-04-24T01:59:39.168 INFO:tasks.rados.rados.0.smithi079.stdout:1106: left oid 40 (ObjNum 545 snap 123 seq_num 545) 2024-04-24T01:59:39.168 INFO:tasks.rados.rados.0.smithi079.stdout:1108: finishing write tid 1 to smithi07929762-13 2024-04-24T01:59:39.168 INFO:tasks.rados.rados.0.smithi079.stdout:1108: finishing write tid 2 to smithi07929762-13 2024-04-24T01:59:39.168 INFO:tasks.rados.rados.0.smithi079.stdout:1108: finishing write tid 3 to smithi07929762-13 2024-04-24T01:59:39.168 INFO:tasks.rados.rados.0.smithi079.stdout:1108: finishing write tid 4 to smithi07929762-13 2024-04-24T01:59:39.168 INFO:tasks.rados.rados.0.smithi079.stdout:1108: finishing write tid 5 to smithi07929762-13 2024-04-24T01:59:39.168 INFO:tasks.rados.rados.0.smithi079.stdout:1108: oid 13 updating version 0 to 385 2024-04-24T01:59:39.168 INFO:tasks.rados.rados.0.smithi079.stdout:1108: oid 13 version 385 is already newer than 385 2024-04-24T01:59:39.168 INFO:tasks.rados.rados.0.smithi079.stdout:1108: oid 13 version 385 is already newer than 382 2024-04-24T01:59:39.168 INFO:tasks.rados.rados.0.smithi079.stdout:1108: oid 13 version 385 is already newer than 385 2024-04-24T01:59:39.168 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 13 v 385 (ObjNum 547 snap 123 seq_num 547) dirty exists 2024-04-24T01:59:39.169 INFO:tasks.rados.rados.0.smithi079.stdout:1108: left oid 13 (ObjNum 547 snap 123 seq_num 547) 2024-04-24T01:59:39.169 INFO:tasks.rados.rados.0.smithi079.stdout:1109: finishing write tid 1 to smithi07929762-43 2024-04-24T01:59:39.169 INFO:tasks.rados.rados.0.smithi079.stdout:1107: finishing write tid 1 to smithi07929762-48 2024-04-24T01:59:39.169 INFO:tasks.rados.rados.0.smithi079.stdout:1107: finishing write tid 2 to smithi07929762-48 2024-04-24T01:59:39.169 INFO:tasks.rados.rados.0.smithi079.stdout:1107: finishing write tid 3 to smithi07929762-48 2024-04-24T01:59:39.169 INFO:tasks.rados.rados.0.smithi079.stdout:1109: finishing write tid 2 to smithi07929762-43 2024-04-24T01:59:39.169 INFO:tasks.rados.rados.0.smithi079.stdout:1109: finishing write tid 3 to smithi07929762-43 2024-04-24T01:59:39.169 INFO:tasks.rados.rados.0.smithi079.stdout:1107: finishing write tid 4 to smithi07929762-48 2024-04-24T01:59:39.169 INFO:tasks.rados.rados.0.smithi079.stdout:1107: finishing write tid 5 to smithi07929762-48 2024-04-24T01:59:39.169 INFO:tasks.rados.rados.0.smithi079.stdout:1107: oid 48 updating version 0 to 185 2024-04-24T01:59:39.169 INFO:tasks.rados.rados.0.smithi079.stdout:1107: oid 48 updating version 185 to 188 2024-04-24T01:59:39.169 INFO:tasks.rados.rados.0.smithi079.stdout:1107: oid 48 version 188 is already newer than 188 2024-04-24T01:59:39.169 INFO:tasks.rados.rados.0.smithi079.stdout:1107: oid 48 version 188 is already newer than 188 2024-04-24T01:59:39.169 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 48 v 188 (ObjNum 546 snap 123 seq_num 546) dirty exists 2024-04-24T01:59:39.169 INFO:tasks.rados.rados.0.smithi079.stdout:1107: left oid 48 (ObjNum 546 snap 123 seq_num 546) 2024-04-24T01:59:39.169 INFO:tasks.rados.rados.0.smithi079.stdout:1109: finishing write tid 4 to smithi07929762-43 2024-04-24T01:59:39.170 INFO:tasks.rados.rados.0.smithi079.stdout:1109: finishing write tid 5 to smithi07929762-43 2024-04-24T01:59:39.170 INFO:tasks.rados.rados.0.smithi079.stdout:1109: oid 43 updating version 0 to 129 2024-04-24T01:59:39.170 INFO:tasks.rados.rados.0.smithi079.stdout:1109: oid 43 version 129 is already newer than 126 2024-04-24T01:59:39.170 INFO:tasks.rados.rados.0.smithi079.stdout:1109: oid 43 version 129 is already newer than 129 2024-04-24T01:59:39.170 INFO:tasks.rados.rados.0.smithi079.stdout:1109: oid 43 version 129 is already newer than 129 2024-04-24T01:59:39.170 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 43 v 129 (ObjNum 548 snap 123 seq_num 548) dirty exists 2024-04-24T01:59:39.170 INFO:tasks.rados.rados.0.smithi079.stdout:1109: left oid 43 (ObjNum 548 snap 123 seq_num 548) 2024-04-24T01:59:39.265 INFO:tasks.rados.rados.0.smithi079.stdout:1111: finishing write tid 1 to smithi07929762-3 2024-04-24T01:59:39.283 INFO:tasks.rados.rados.0.smithi079.stdout:1111: finishing write tid 2 to smithi07929762-3 2024-04-24T01:59:39.291 INFO:tasks.rados.rados.0.smithi079.stdout:1111: finishing write tid 3 to smithi07929762-3 2024-04-24T01:59:39.296 INFO:tasks.rados.rados.0.smithi079.stdout:1111: finishing write tid 4 to smithi07929762-3 2024-04-24T01:59:39.297 INFO:tasks.rados.rados.0.smithi079.stdout:1111: finishing write tid 5 to smithi07929762-3 2024-04-24T01:59:39.297 INFO:tasks.rados.rados.0.smithi079.stdout:1111: oid 3 updating version 0 to 527 2024-04-24T01:59:39.297 INFO:tasks.rados.rados.0.smithi079.stdout:1111: oid 3 updating version 527 to 530 2024-04-24T01:59:39.297 INFO:tasks.rados.rados.0.smithi079.stdout:1111: oid 3 version 530 is already newer than 530 2024-04-24T01:59:39.297 INFO:tasks.rados.rados.0.smithi079.stdout:1111: oid 3 version 530 is already newer than 530 2024-04-24T01:59:39.298 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 3 v 530 (ObjNum 549 snap 123 seq_num 549) dirty exists 2024-04-24T01:59:39.298 INFO:tasks.rados.rados.0.smithi079.stdout:1111: left oid 3 (ObjNum 549 snap 123 seq_num 549) 2024-04-24T01:59:40.013 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T01:59:40.126 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:59:40.124+0000 7fd0c5e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:59:40.130 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:59:40.129+0000 7fd0c5e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:59:40.152 INFO:tasks.rados.rados.0.smithi079.stdout:1106: done (5 left) 2024-04-24T01:59:40.152 INFO:tasks.rados.rados.0.smithi079.stdout:1107: done (4 left) 2024-04-24T01:59:40.152 INFO:tasks.rados.rados.0.smithi079.stdout:1108: done (3 left) 2024-04-24T01:59:40.152 INFO:tasks.rados.rados.0.smithi079.stdout:1109: done (2 left) 2024-04-24T01:59:40.152 INFO:tasks.rados.rados.0.smithi079.stdout:1111: done (1 left) 2024-04-24T01:59:40.152 INFO:tasks.rados.rados.0.smithi079.stdout:1112: done (0 left) 2024-04-24T01:59:40.153 INFO:tasks.rados.rados.0.smithi079.stdout:1113: snap_create 2024-04-24T01:59:40.921 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-04-24T01:59:40.921 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-04-24T01:59:40.922 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T01:59:41.034 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:59:41.030+0000 7f8003c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:59:41.038 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:59:41.037+0000 7f8003c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:59:41.053 INFO:tasks.rados.rados.0.smithi079.stdout:1113: done (0 left) 2024-04-24T01:59:41.053 INFO:tasks.rados.rados.0.smithi079.stdout:1114: read oid 17 snap -1 2024-04-24T01:59:41.053 INFO:tasks.rados.rados.0.smithi079.stdout:1114: expect deleted 2024-04-24T01:59:41.053 INFO:tasks.rados.rados.0.smithi079.stdout:1115: write oid 12 current snap is 125 2024-04-24T01:59:41.053 INFO:tasks.rados.rados.0.smithi079.stdout:1115: seq_num 550 ranges {443850=564366,1634861=732856,2770646=473204} 2024-04-24T01:59:41.060 INFO:tasks.rados.rados.0.smithi079.stdout:1115: writing smithi07929762-12 from 443850 to 1008216 tid 1 2024-04-24T01:59:41.067 INFO:tasks.rados.rados.0.smithi079.stdout:1115: writing smithi07929762-12 from 1634861 to 2367717 tid 2 2024-04-24T01:59:41.073 INFO:tasks.rados.rados.0.smithi079.stdout:1115: writing smithi07929762-12 from 2770646 to 3243850 tid 3 2024-04-24T01:59:41.073 INFO:tasks.rados.rados.0.smithi079.stdout:1116: read oid 7 snap -1 2024-04-24T01:59:41.073 INFO:tasks.rados.rados.0.smithi079.stdout:1116: expect (ObjNum 541 snap 121 seq_num 541) 2024-04-24T01:59:41.073 INFO:tasks.rados.rados.0.smithi079.stdout:1117: snap_remove snap 121 2024-04-24T01:59:41.250 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:59:41.250 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-24T01:59:41.262 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":389,"stamp":"2024-04-24T01:59:41.050148+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":90934705,"num_objects":60,"num_object_clones":21,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":5,"num_read":2104,"num_read_kb":1413360,"num_write":1948,"num_write_kb":493243,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3707,"ondisk_log_size":3707,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":376896,"kb_used_data":236320,"kb_used_omap":0,"kb_used_meta":140544,"kb_avail":377110464,"statfs":{"total":386547056640,"available":386161115136,"internally_reserved":0,"allocated":241991680,"data_stored":258816675,"data_compressed":89889,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":143917056},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":7668064,"num_objects":7,"num_object_clones":5,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":40,"num_read_kb":23152,"num_write":45,"num_write_kb":12971,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.924611"},"pg_stats":[{"pgid":"3.f","version":"261'188","reported_seq":983,"reported_epoch":262,"state":"active+clean+snaptrim","last_fresh":"2024-04-24T01:59:39.170470+0000","last_change":"2024-04-24T01:59:39.170470+0000","last_active":"2024-04-24T01:59:39.170470+0000","last_peered":"2024-04-24T01:59:39.170470+0000","last_clean":"2024-04-24T01:59:39.170470+0000","last_became_active":"2024-04-24T01:55:30.386726+0000","last_became_peered":"2024-04-24T01:55:30.386726+0000","last_unstale":"2024-04-24T01:59:39.170470+0000","last_undegraded":"2024-04-24T01:59:39.170470+0000","last_fullsized":"2024-04-24T01:59:39.170470+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":188,"log_dups_size":0,"ondisk_log_size":188,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":17,"snaptrim_duration":0,"stat_sum":{"num_bytes":5462440,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":148,"num_read_kb":129899,"num_write":93,"num_write_kb":26702,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"260'309","reported_seq":1044,"reported_epoch":262,"state":"active+clean+snaptrim","last_fresh":"2024-04-24T01:59:39.161148+0000","last_change":"2024-04-24T01:59:39.161148+0000","last_active":"2024-04-24T01:59:39.161148+0000","last_peered":"2024-04-24T01:59:39.161148+0000","last_clean":"2024-04-24T01:59:39.161148+0000","last_became_active":"2024-04-24T01:55:30.321866+0000","last_became_peered":"2024-04-24T01:55:30.321866+0000","last_unstale":"2024-04-24T01:59:39.161148+0000","last_undegraded":"2024-04-24T01:59:39.161148+0000","last_fullsized":"2024-04-24T01:59:39.161148+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":309,"log_dups_size":0,"ondisk_log_size":309,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":27,"snaptrim_duration":0,"stat_sum":{"num_bytes":5143140,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":165,"num_read_kb":113141,"num_write":151,"num_write_kb":40824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"257'227","reported_seq":978,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-04-24T01:59:38.139710+0000","last_change":"2024-04-24T01:59:37.156035+0000","last_active":"2024-04-24T01:59:38.139710+0000","last_peered":"2024-04-24T01:59:38.139710+0000","last_clean":"2024-04-24T01:59:38.139710+0000","last_became_active":"2024-04-24T01:55:30.320186+0000","last_became_peered":"2024-04-24T01:55:30.320186+0000","last_unstale":"2024-04-24T01:59:38.139710+0000","last_undegraded":"2024-04-24T01:59:38.139710+0000","last_fullsized":"2024-04-24T01:59:38.139710+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":227,"log_dups_size":0,"ondisk_log_size":227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":22,"snaptrim_duration":0,"stat_sum":{"num_bytes":5159914,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":125,"num_read_kb":85296,"num_write":124,"num_write_kb":34018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"246'263","reported_seq":1027,"reported_epoch":262,"state":"active+clean+snaptrim","last_fresh":"2024-04-24T01:59:39.173727+0000","last_change":"2024-04-24T01:59:39.173727+0000","last_active":"2024-04-24T01:59:39.173727+0000","last_peered":"2024-04-24T01:59:39.173727+0000","last_clean":"2024-04-24T01:59:39.173727+0000","last_became_active":"2024-04-24T01:55:30.288081+0000","last_became_peered":"2024-04-24T01:55:30.288081+0000","last_unstale":"2024-04-24T01:59:39.173727+0000","last_undegraded":"2024-04-24T01:59:39.173727+0000","last_fullsized":"2024-04-24T01:59:39.173727+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":263,"log_dups_size":0,"ondisk_log_size":263,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":27,"snaptrim_duration":0,"stat_sum":{"num_bytes":2269044,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":136,"num_read_kb":92553,"num_write":129,"num_write_kb":33953,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"260'99","reported_seq":856,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-04-24T01:59:38.250189+0000","last_change":"2024-04-24T01:59:37.253542+0000","last_active":"2024-04-24T01:59:38.250189+0000","last_peered":"2024-04-24T01:59:38.250189+0000","last_clean":"2024-04-24T01:59:38.250189+0000","last_became_active":"2024-04-24T01:55:30.333815+0000","last_became_peered":"2024-04-24T01:55:30.333815+0000","last_unstale":"2024-04-24T01:59:38.250189+0000","last_undegraded":"2024-04-24T01:59:38.250189+0000","last_fullsized":"2024-04-24T01:59:38.250189+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":10,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":45,"num_read_kb":23146,"num_write":56,"num_write_kb":15361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"261'521","reported_seq":1244,"reported_epoch":262,"state":"active+clean+snaptrim","last_fresh":"2024-04-24T01:59:39.172747+0000","last_change":"2024-04-24T01:59:39.172747+0000","last_active":"2024-04-24T01:59:39.172747+0000","last_peered":"2024-04-24T01:59:39.172747+0000","last_clean":"2024-04-24T01:59:39.172747+0000","last_became_active":"2024-04-24T01:55:30.315075+0000","last_became_peered":"2024-04-24T01:55:30.315075+0000","last_unstale":"2024-04-24T01:59:39.172747+0000","last_undegraded":"2024-04-24T01:59:39.172747+0000","last_fullsized":"2024-04-24T01:59:39.172747+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":521,"log_dups_size":0,"ondisk_log_size":521,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":47,"snaptrim_duration":0,"stat_sum":{"num_bytes":16488261,"num_objects":9,"num_object_clones":3,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":243,"num_read_kb":153595,"num_write":240,"num_write_kb":62809,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"261'95","reported_seq":876,"reported_epoch":262,"state":"active+clean+snaptrim","last_fresh":"2024-04-24T01:59:39.181870+0000","last_change":"2024-04-24T01:59:39.181870+0000","last_active":"2024-04-24T01:59:39.181870+0000","last_peered":"2024-04-24T01:59:39.181870+0000","last_clean":"2024-04-24T01:59:39.181870+0000","last_became_active":"2024-04-24T01:55:30.350300+0000","last_became_peered":"2024-04-24T01:55:30.350300+0000","last_unstale":"2024-04-24T01:59:39.181870+0000","last_undegraded":"2024-04-24T01:59:39.181870+0000","last_fullsized":"2024-04-24T01:59:39.181870+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":8,"snaptrim_duration":0,"stat_sum":{"num_bytes":3916097,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":46,"num_read_kb":22974,"num_write":62,"num_write_kb":15453,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"257'32","reported_seq":794,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-04-24T01:59:38.150611+0000","last_change":"2024-04-24T01:59:37.166970+0000","last_active":"2024-04-24T01:59:38.150611+0000","last_peered":"2024-04-24T01:59:38.150611+0000","last_clean":"2024-04-24T01:59:38.150611+0000","last_became_active":"2024-04-24T01:55:30.376813+0000","last_became_peered":"2024-04-24T01:55:30.376813+0000","last_unstale":"2024-04-24T01:59:38.150611+0000","last_undegraded":"2024-04-24T01:59:38.150611+0000","last_fullsized":"2024-04-24T01:59:38.150611+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0,"stat_sum":{"num_bytes":3025359,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":10082,"num_write":27,"num_write_kb":6802,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'97","reported_seq":789,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-04-24T01:59:38.242226+0000","last_change":"2024-04-24T01:55:16.196470+0000","last_active":"2024-04-24T01:59:38.242226+0000","last_peered":"2024-04-24T01:59:38.242226+0000","last_clean":"2024-04-24T01:59:38.242226+0000","last_became_active":"2024-04-24T01:55:16.190464+0000","last_became_peered":"2024-04-24T01:55:16.190464+0000","last_unstale":"2024-04-24T01:59:38.242226+0000","last_undegraded":"2024-04-24T01:59:38.242226+0000","last_fullsized":"2024-04-24T01:59:38.242226+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:15.127115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:15.127115+0000","last_clean_scrub_stamp":"2024-04-24T01:55:15.127115+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":652,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-04-24T01:59:38.150055+0000","last_change":"2024-04-24T01:55:18.236148+0000","last_active":"2024-04-24T01:59:38.150055+0000","last_peered":"2024-04-24T01:59:38.150055+0000","last_clean":"2024-04-24T01:59:38.150055+0000","last_became_active":"2024-04-24T01:55:18.230154+0000","last_became_peered":"2024-04-24T01:55:18.230154+0000","last_unstale":"2024-04-24T01:59:38.150055+0000","last_undegraded":"2024-04-24T01:59:38.150055+0000","last_fullsized":"2024-04-24T01:59:38.150055+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"260'381","reported_seq":1133,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-04-24T01:59:38.148687+0000","last_change":"2024-04-24T01:59:37.231720+0000","last_active":"2024-04-24T01:59:38.148687+0000","last_peered":"2024-04-24T01:59:38.148687+0000","last_clean":"2024-04-24T01:59:38.148687+0000","last_became_active":"2024-04-24T01:55:30.316222+0000","last_became_peered":"2024-04-24T01:55:30.316222+0000","last_unstale":"2024-04-24T01:59:38.148687+0000","last_undegraded":"2024-04-24T01:59:38.148687+0000","last_fullsized":"2024-04-24T01:59:38.148687+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":381,"log_dups_size":0,"ondisk_log_size":381,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":29,"snaptrim_duration":0,"stat_sum":{"num_bytes":10558967,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":211,"num_read_kb":149106,"num_write":201,"num_write_kb":56555,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":673,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-04-24T01:59:39.181536+0000","last_change":"2024-04-24T01:55:18.265905+0000","last_active":"2024-04-24T01:59:39.181536+0000","last_peered":"2024-04-24T01:59:39.181536+0000","last_clean":"2024-04-24T01:59:39.181536+0000","last_became_active":"2024-04-24T01:55:18.257020+0000","last_became_peered":"2024-04-24T01:55:18.257020+0000","last_unstale":"2024-04-24T01:59:39.181536+0000","last_undegraded":"2024-04-24T01:59:39.181536+0000","last_fullsized":"2024-04-24T01:59:39.181536+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"260'317","reported_seq":1042,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-04-24T01:59:38.244768+0000","last_change":"2024-04-24T01:59:37.231073+0000","last_active":"2024-04-24T01:59:38.244768+0000","last_peered":"2024-04-24T01:59:38.244768+0000","last_clean":"2024-04-24T01:59:38.244768+0000","last_became_active":"2024-04-24T01:55:30.344129+0000","last_became_peered":"2024-04-24T01:55:30.344129+0000","last_unstale":"2024-04-24T01:59:38.244768+0000","last_undegraded":"2024-04-24T01:59:38.244768+0000","last_fullsized":"2024-04-24T01:59:38.244768+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":317,"log_dups_size":0,"ondisk_log_size":317,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":26,"snaptrim_duration":0,"stat_sum":{"num_bytes":3042166,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":133,"num_read_kb":72539,"num_write":149,"num_write_kb":41796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":652,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-04-24T01:59:38.138652+0000","last_change":"2024-04-24T01:55:18.215251+0000","last_active":"2024-04-24T01:59:38.138652+0000","last_peered":"2024-04-24T01:59:38.138652+0000","last_clean":"2024-04-24T01:59:38.138652+0000","last_became_active":"2024-04-24T01:55:18.206605+0000","last_became_peered":"2024-04-24T01:55:18.206605+0000","last_unstale":"2024-04-24T01:59:38.138652+0000","last_undegraded":"2024-04-24T01:59:38.138652+0000","last_fullsized":"2024-04-24T01:59:38.138652+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"247'144","reported_seq":923,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-04-24T01:59:38.239815+0000","last_change":"2024-04-24T01:59:37.210049+0000","last_active":"2024-04-24T01:59:38.239815+0000","last_peered":"2024-04-24T01:59:38.239815+0000","last_clean":"2024-04-24T01:59:38.239815+0000","last_became_active":"2024-04-24T01:55:30.401366+0000","last_became_peered":"2024-04-24T01:55:30.401366+0000","last_unstale":"2024-04-24T01:59:38.239815+0000","last_undegraded":"2024-04-24T01:59:38.239815+0000","last_fullsized":"2024-04-24T01:59:38.239815+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":144,"log_dups_size":0,"ondisk_log_size":144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":11,"snaptrim_duration":0,"stat_sum":{"num_bytes":5563282,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":130,"num_read_kb":104968,"num_write":55,"num_write_kb":14247,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":673,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-04-24T01:59:39.187060+0000","last_change":"2024-04-24T01:55:18.230893+0000","last_active":"2024-04-24T01:59:39.187060+0000","last_peered":"2024-04-24T01:59:39.187060+0000","last_clean":"2024-04-24T01:59:39.187060+0000","last_became_active":"2024-04-24T01:55:18.225188+0000","last_became_peered":"2024-04-24T01:55:18.225188+0000","last_unstale":"2024-04-24T01:59:39.187060+0000","last_undegraded":"2024-04-24T01:59:39.187060+0000","last_fullsized":"2024-04-24T01:59:39.187060+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"260'251","reported_seq":983,"reported_epoch":262,"state":"active+clean+snaptrim","last_fresh":"2024-04-24T01:59:39.162552+0000","last_change":"2024-04-24T01:59:39.162552+0000","last_active":"2024-04-24T01:59:39.162552+0000","last_peered":"2024-04-24T01:59:39.162552+0000","last_clean":"2024-04-24T01:59:39.162552+0000","last_became_active":"2024-04-24T01:55:30.343318+0000","last_became_peered":"2024-04-24T01:55:30.343318+0000","last_unstale":"2024-04-24T01:59:39.162552+0000","last_undegraded":"2024-04-24T01:59:39.162552+0000","last_fullsized":"2024-04-24T01:59:39.162552+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":251,"log_dups_size":0,"ondisk_log_size":251,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":23,"snaptrim_duration":0,"stat_sum":{"num_bytes":3092587,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":133,"num_read_kb":107523,"num_write":113,"num_write_kb":31479,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"261'330","reported_seq":1073,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-04-24T01:59:38.184830+0000","last_change":"2024-04-24T01:59:37.231145+0000","last_active":"2024-04-24T01:59:38.184830+0000","last_peered":"2024-04-24T01:59:38.184830+0000","last_clean":"2024-04-24T01:59:38.184830+0000","last_became_active":"2024-04-24T01:55:30.345308+0000","last_became_peered":"2024-04-24T01:55:30.345308+0000","last_unstale":"2024-04-24T01:59:38.184830+0000","last_undegraded":"2024-04-24T01:59:38.184830+0000","last_fullsized":"2024-04-24T01:59:38.184830+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":330,"log_dups_size":0,"ondisk_log_size":330,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":27,"snaptrim_duration":0,"stat_sum":{"num_bytes":8336569,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":177,"num_read_kb":114667,"num_write":160,"num_write_kb":42749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":641,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-04-24T01:59:38.335211+0000","last_change":"2024-04-24T01:55:18.224238+0000","last_active":"2024-04-24T01:59:38.335211+0000","last_peered":"2024-04-24T01:59:38.335211+0000","last_clean":"2024-04-24T01:59:38.335211+0000","last_became_active":"2024-04-24T01:55:18.219131+0000","last_became_peered":"2024-04-24T01:55:18.219131+0000","last_unstale":"2024-04-24T01:59:38.335211+0000","last_undegraded":"2024-04-24T01:59:38.335211+0000","last_fullsized":"2024-04-24T01:59:38.335211+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"260'66","reported_seq":823,"reported_epoch":262,"state":"active+clean+snaptrim","last_fresh":"2024-04-24T01:59:39.179309+0000","last_change":"2024-04-24T01:59:39.179309+0000","last_active":"2024-04-24T01:59:39.179309+0000","last_peered":"2024-04-24T01:59:39.179309+0000","last_clean":"2024-04-24T01:59:39.179309+0000","last_became_active":"2024-04-24T01:55:30.319958+0000","last_became_peered":"2024-04-24T01:55:30.319958+0000","last_unstale":"2024-04-24T01:59:39.179309+0000","last_undegraded":"2024-04-24T01:59:39.179309+0000","last_fullsized":"2024-04-24T01:59:39.179309+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":5,"snaptrim_duration":0,"stat_sum":{"num_bytes":3126201,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":40,"num_read_kb":36473,"num_write":46,"num_write_kb":13159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":656,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-04-24T01:59:38.237870+0000","last_change":"2024-04-24T01:55:18.235763+0000","last_active":"2024-04-24T01:59:38.237870+0000","last_peered":"2024-04-24T01:59:38.237870+0000","last_clean":"2024-04-24T01:59:38.237870+0000","last_became_active":"2024-04-24T01:55:18.229175+0000","last_became_peered":"2024-04-24T01:55:18.229175+0000","last_unstale":"2024-04-24T01:59:38.237870+0000","last_undegraded":"2024-04-24T01:59:38.237870+0000","last_fullsized":"2024-04-24T01:59:38.237870+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"261'129","reported_seq":881,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-04-24T01:59:38.326907+0000","last_change":"2024-04-24T01:59:37.248495+0000","last_active":"2024-04-24T01:59:38.326907+0000","last_peered":"2024-04-24T01:59:38.326907+0000","last_clean":"2024-04-24T01:59:38.326907+0000","last_became_active":"2024-04-24T01:55:30.372690+0000","last_became_peered":"2024-04-24T01:55:30.372690+0000","last_unstale":"2024-04-24T01:59:38.326907+0000","last_undegraded":"2024-04-24T01:59:38.326907+0000","last_fullsized":"2024-04-24T01:59:38.326907+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":9,"snaptrim_duration":0,"stat_sum":{"num_bytes":6067525,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":60,"num_read_kb":39375,"num_write":63,"num_write_kb":18028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":643,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-04-24T01:59:39.178290+0000","last_change":"2024-04-24T01:55:18.278351+0000","last_active":"2024-04-24T01:59:39.178290+0000","last_peered":"2024-04-24T01:59:39.178290+0000","last_clean":"2024-04-24T01:59:39.178290+0000","last_became_active":"2024-04-24T01:55:18.273668+0000","last_became_peered":"2024-04-24T01:55:18.273668+0000","last_unstale":"2024-04-24T01:59:39.178290+0000","last_undegraded":"2024-04-24T01:59:39.178290+0000","last_fullsized":"2024-04-24T01:59:39.178290+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"260'258","reported_seq":1027,"reported_epoch":262,"state":"active+clean+snaptrim","last_fresh":"2024-04-24T01:59:39.176959+0000","last_change":"2024-04-24T01:59:39.176959+0000","last_active":"2024-04-24T01:59:39.176959+0000","last_peered":"2024-04-24T01:59:39.176959+0000","last_clean":"2024-04-24T01:59:39.176959+0000","last_became_active":"2024-04-24T01:55:30.363123+0000","last_became_peered":"2024-04-24T01:55:30.363123+0000","last_unstale":"2024-04-24T01:59:39.176959+0000","last_undegraded":"2024-04-24T01:59:39.176959+0000","last_fullsized":"2024-04-24T01:59:39.176959+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":258,"log_dups_size":0,"ondisk_log_size":258,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":27,"snaptrim_duration":0,"stat_sum":{"num_bytes":9092785,"num_objects":6,"num_object_clones":2,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":194,"num_read_kb":157932,"num_write":142,"num_write_kb":37414,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":652,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-04-24T01:59:38.147762+0000","last_change":"2024-04-24T01:55:18.241117+0000","last_active":"2024-04-24T01:59:38.147762+0000","last_peered":"2024-04-24T01:59:38.147762+0000","last_clean":"2024-04-24T01:59:38.147762+0000","last_became_active":"2024-04-24T01:55:18.234313+0000","last_became_peered":"2024-04-24T01:55:18.234313+0000","last_unstale":"2024-04-24T01:59:38.147762+0000","last_undegraded":"2024-04-24T01:59:38.147762+0000","last_fullsized":"2024-04-24T01:59:38.147762+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":90344337,"num_objects":58,"num_object_clones":21,"num_object_copies":174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":5,"num_read":2001,"num_read_kb":1413269,"num_write":1811,"num_write_kb":491349,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":235491328,"data_stored":255540637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3610,"ondisk_log_size":3610,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":1}],"osd_stats":[{"osd":3,"up_from":10,"seq":42949673224,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":100784,"kb_used_data":65640,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94271056,"statfs":{"total":96636764160,"available":96533561344,"internally_reserved":0,"allocated":67215360,"data_stored":71458526,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673224,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":86468,"kb_used_data":51324,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94285372,"statfs":{"total":96636764160,"available":96548220928,"internally_reserved":0,"allocated":52555776,"data_stored":55955030,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705929,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":88580,"kb_used_data":53436,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94283260,"statfs":{"total":96636764160,"available":96546058240,"internally_reserved":0,"allocated":54718464,"data_stored":59886278,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705928,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":101064,"kb_used_data":65920,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94270776,"statfs":{"total":96636764160,"available":96533274624,"internally_reserved":0,"allocated":67502080,"data_stored":71516841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":52940800,"data_stored":58785520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":67186688,"data_stored":72389203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":50352128,"data_stored":54431209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65011712,"data_stored":69934705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T01:59:42.055 INFO:tasks.rados.rados.0.smithi079.stdout:1117: done (3 left) 2024-04-24T01:59:42.055 INFO:tasks.rados.rados.0.smithi079.stdout:1118: snap_create 2024-04-24T01:59:42.055 INFO:tasks.rados.rados.0.smithi079.stdout:1116: expect (ObjNum 541 snap 121 seq_num 541) 2024-04-24T01:59:42.230 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T01:59:42.247 INFO:tasks.rados.rados.0.smithi079.stdout:1115: finishing write tid 1 to smithi07929762-12 2024-04-24T01:59:42.247 INFO:tasks.rados.rados.0.smithi079.stdout:1115: finishing write tid 2 to smithi07929762-12 2024-04-24T01:59:42.247 INFO:tasks.rados.rados.0.smithi079.stdout:1115: finishing write tid 3 to smithi07929762-12 2024-04-24T01:59:42.247 INFO:tasks.rados.rados.0.smithi079.stdout:1115: finishing write tid 4 to smithi07929762-12 2024-04-24T01:59:42.247 INFO:tasks.rados.rados.0.smithi079.stdout:1115: finishing write tid 5 to smithi07929762-12 2024-04-24T01:59:42.247 INFO:tasks.rados.rados.0.smithi079.stdout:1115: oid 12 updating version 0 to 262 2024-04-24T01:59:42.247 INFO:tasks.rados.rados.0.smithi079.stdout:1115: oid 12 version 262 is already newer than 262 2024-04-24T01:59:42.247 INFO:tasks.rados.rados.0.smithi079.stdout:1115: oid 12 version 262 is already newer than 262 2024-04-24T01:59:42.247 INFO:tasks.rados.rados.0.smithi079.stdout:1115: oid 12 version 262 is already newer than 262 2024-04-24T01:59:42.247 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 12 v 262 (ObjNum 550 snap 125 seq_num 550) dirty exists 2024-04-24T01:59:42.247 INFO:tasks.rados.rados.0.smithi079.stdout:1115: left oid 12 (ObjNum 550 snap 125 seq_num 550) 2024-04-24T01:59:42.342 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:59:42.339+0000 7f87ab75d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:59:42.345 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:59:42.345+0000 7f87ab75d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:59:43.062 INFO:tasks.rados.rados.0.smithi079.stdout:1114: done (3 left) 2024-04-24T01:59:43.062 INFO:tasks.rados.rados.0.smithi079.stdout:1115: done (2 left) 2024-04-24T01:59:43.063 INFO:tasks.rados.rados.0.smithi079.stdout:1116: done (1 left) 2024-04-24T01:59:43.063 INFO:tasks.rados.rados.0.smithi079.stdout:1118: done (0 left) 2024-04-24T01:59:43.063 INFO:tasks.rados.rados.0.smithi079.stdout:1119: write oid 1 current snap is 126 2024-04-24T01:59:43.063 INFO:tasks.rados.rados.0.smithi079.stdout:1119: seq_num 551 ranges {460657=639615,1693343=476514,2681716=578941} 2024-04-24T01:59:43.068 INFO:tasks.rados.rados.0.smithi079.stdout:1119: writing smithi07929762-1 from 460657 to 1100272 tid 1 2024-04-24T01:59:43.074 INFO:tasks.rados.rados.0.smithi079.stdout:1119: writing smithi07929762-1 from 1693343 to 2169857 tid 2 2024-04-24T01:59:43.081 INFO:tasks.rados.rados.0.smithi079.stdout:1119: writing smithi07929762-1 from 2681716 to 3260657 tid 3 2024-04-24T01:59:43.081 INFO:tasks.rados.rados.0.smithi079.stdout:1120: delete oid 30 current snap is 126 2024-04-24T01:59:43.141 INFO:tasks.rados.rados.0.smithi079.stdout:1119: finishing write tid 1 to smithi07929762-1 2024-04-24T01:59:43.163 INFO:tasks.rados.rados.0.smithi079.stdout:1119: finishing write tid 2 to smithi07929762-1 2024-04-24T01:59:43.179 INFO:tasks.rados.rados.0.smithi079.stdout:1120: done (1 left) 2024-04-24T01:59:43.179 INFO:tasks.rados.rados.0.smithi079.stdout:1121: read oid 24 snap -1 2024-04-24T01:59:43.180 INFO:tasks.rados.rados.0.smithi079.stdout:1121: expect deleted 2024-04-24T01:59:43.180 INFO:tasks.rados.rados.0.smithi079.stdout:1122: snap_remove snap 123 2024-04-24T01:59:44.061 INFO:tasks.rados.rados.0.smithi079.stdout:1122: done (2 left) 2024-04-24T01:59:44.061 INFO:tasks.rados.rados.0.smithi079.stdout:1123: snap_create 2024-04-24T01:59:44.061 INFO:tasks.rados.rados.0.smithi079.stdout:1119: finishing write tid 3 to smithi07929762-1 2024-04-24T01:59:44.061 INFO:tasks.rados.rados.0.smithi079.stdout:1119: finishing write tid 4 to smithi07929762-1 2024-04-24T01:59:44.061 INFO:tasks.rados.rados.0.smithi079.stdout:1119: finishing write tid 5 to smithi07929762-1 2024-04-24T01:59:44.061 INFO:tasks.rados.rados.0.smithi079.stdout:1119: oid 1 updating version 0 to 537 2024-04-24T01:59:44.061 INFO:tasks.rados.rados.0.smithi079.stdout:1119: oid 1 version 537 is already newer than 537 2024-04-24T01:59:44.061 INFO:tasks.rados.rados.0.smithi079.stdout:1119: oid 1 version 537 is already newer than 537 2024-04-24T01:59:44.061 INFO:tasks.rados.rados.0.smithi079.stdout:1119: oid 1 version 537 is already newer than 537 2024-04-24T01:59:44.061 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 1 v 537 (ObjNum 551 snap 126 seq_num 551) dirty exists 2024-04-24T01:59:44.061 INFO:tasks.rados.rados.0.smithi079.stdout:1119: left oid 1 (ObjNum 551 snap 126 seq_num 551) 2024-04-24T01:59:44.438 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T01:59:44.545 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:59:44.542+0000 7fafa9cad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:59:44.548 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:59:44.548+0000 7fafa9cad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:59:45.059 INFO:tasks.rados.rados.0.smithi079.stdout:1119: done (2 left) 2024-04-24T01:59:45.059 INFO:tasks.rados.rados.0.smithi079.stdout:1121: done (1 left) 2024-04-24T01:59:45.060 INFO:tasks.rados.rados.0.smithi079.stdout:1123: done (0 left) 2024-04-24T01:59:45.060 INFO:tasks.rados.rados.0.smithi079.stdout:1124: write oid 48 current snap is 127 2024-04-24T01:59:45.060 INFO:tasks.rados.rados.0.smithi079.stdout:1124: seq_num 553 ranges {494271=790113,1693954=480183,2620209=512129} 2024-04-24T01:59:45.066 INFO:tasks.rados.rados.0.smithi079.stdout:1124: writing smithi07929762-48 from 494271 to 1284384 tid 1 2024-04-24T01:59:45.072 INFO:tasks.rados.rados.0.smithi079.stdout:1124: writing smithi07929762-48 from 1693954 to 2174137 tid 2 2024-04-24T01:59:45.082 INFO:tasks.rados.rados.0.smithi079.stdout:1124: writing smithi07929762-48 from 2620209 to 3132338 tid 3 2024-04-24T01:59:45.082 INFO:tasks.rados.rados.0.smithi079.stdout:1125: snap_remove snap 122 2024-04-24T01:59:46.073 INFO:tasks.rados.rados.0.smithi079.stdout:1125: done (1 left) 2024-04-24T01:59:46.074 INFO:tasks.rados.rados.0.smithi079.stdout:1126: read oid 12 snap 124 2024-04-24T01:59:46.074 INFO:tasks.rados.rados.0.smithi079.stdout:1126: expect deleted 2024-04-24T01:59:46.074 INFO:tasks.rados.rados.0.smithi079.stdout:1127: write oid 38 current snap is 127 2024-04-24T01:59:46.074 INFO:tasks.rados.rados.0.smithi079.stdout:1124: finishing write tid 1 to smithi07929762-48 2024-04-24T01:59:46.074 INFO:tasks.rados.rados.0.smithi079.stdout:1127: seq_num 554 ranges {511078=465362,1752436=740194,3163985=147093} 2024-04-24T01:59:46.078 INFO:tasks.rados.rados.0.smithi079.stdout:1127: writing smithi07929762-38 from 511078 to 976440 tid 1 2024-04-24T01:59:46.086 INFO:tasks.rados.rados.0.smithi079.stdout:1127: writing smithi07929762-38 from 1752436 to 2492630 tid 2 2024-04-24T01:59:46.091 INFO:tasks.rados.rados.0.smithi079.stdout:1127: writing smithi07929762-38 from 3163985 to 3311078 tid 3 2024-04-24T01:59:46.092 INFO:tasks.rados.rados.0.smithi079.stdout:1128: delete oid 22 current snap is 127 2024-04-24T01:59:46.092 INFO:tasks.rados.rados.0.smithi079.stdout:1124: finishing write tid 2 to smithi07929762-48 2024-04-24T01:59:46.092 INFO:tasks.rados.rados.0.smithi079.stdout:1124: finishing write tid 3 to smithi07929762-48 2024-04-24T01:59:46.092 INFO:tasks.rados.rados.0.smithi079.stdout:1124: finishing write tid 4 to smithi07929762-48 2024-04-24T01:59:46.092 INFO:tasks.rados.rados.0.smithi079.stdout:1124: finishing write tid 5 to smithi07929762-48 2024-04-24T01:59:46.092 INFO:tasks.rados.rados.0.smithi079.stdout:1124: oid 48 updating version 0 to 197 2024-04-24T01:59:46.092 INFO:tasks.rados.rados.0.smithi079.stdout:1124: oid 48 version 197 is already newer than 197 2024-04-24T01:59:46.092 INFO:tasks.rados.rados.0.smithi079.stdout:1124: oid 48 version 197 is already newer than 197 2024-04-24T01:59:46.092 INFO:tasks.rados.rados.0.smithi079.stdout:1124: oid 48 version 197 is already newer than 197 2024-04-24T01:59:46.092 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 48 v 197 (ObjNum 553 snap 127 seq_num 553) dirty exists 2024-04-24T01:59:46.092 INFO:tasks.rados.rados.0.smithi079.stdout:1124: left oid 48 (ObjNum 553 snap 127 seq_num 553) 2024-04-24T01:59:46.201 INFO:tasks.rados.rados.0.smithi079.stdout:1127: finishing write tid 1 to smithi07929762-38 2024-04-24T01:59:46.236 INFO:tasks.rados.rados.0.smithi079.stdout:1124: done (3 left) 2024-04-24T01:59:46.236 INFO:tasks.rados.rados.0.smithi079.stdout:1126: done (2 left) 2024-04-24T01:59:46.236 INFO:tasks.rados.rados.0.smithi079.stdout:1128: done (1 left) 2024-04-24T01:59:46.236 INFO:tasks.rados.rados.0.smithi079.stdout:1129: snap_remove snap 126 2024-04-24T01:59:46.264 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-04-24T01:59:46.264 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-04-24T01:59:46.264 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T01:59:46.405 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:59:46.397+0000 7fb09a31d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:59:46.410 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:59:46.409+0000 7fb09a31d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:59:46.607 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:59:46.607 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-24T01:59:46.620 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":396,"stamp":"2024-04-24T01:59:46.072298+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":94225069,"num_objects":55,"num_object_clones":17,"num_object_copies":165,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":2116,"num_read_kb":1417340,"num_write":1970,"num_write_kb":498834,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3761,"ondisk_log_size":3761,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":356588,"kb_used_data":216012,"kb_used_omap":0,"kb_used_meta":140544,"kb_avail":377130772,"statfs":{"total":386547056640,"available":386181910528,"internally_reserved":0,"allocated":221196288,"data_stored":238003363,"data_compressed":89889,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":143917056},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3290364,"num_objects":-5,"num_object_clones":-4,"num_object_copies":-15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":-1,"num_read":12,"num_read_kb":3980,"num_write":22,"num_write_kb":5591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.019187"},"pg_stats":[{"pgid":"3.f","version":"265'192","reported_seq":999,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-04-24T01:59:44.074808+0000","last_change":"2024-04-24T01:59:44.086107+0000","last_active":"2024-04-24T01:59:44.074808+0000","last_peered":"2024-04-24T01:59:44.074808+0000","last_clean":"2024-04-24T01:59:44.074808+0000","last_became_active":"2024-04-24T01:55:30.386726+0000","last_became_peered":"2024-04-24T01:55:30.386726+0000","last_unstale":"2024-04-24T01:59:44.074808+0000","last_undegraded":"2024-04-24T01:59:44.074808+0000","last_fullsized":"2024-04-24T01:59:44.074808+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":192,"log_dups_size":0,"ondisk_log_size":192,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":18,"snaptrim_duration":0,"stat_sum":{"num_bytes":5462440,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":148,"num_read_kb":129899,"num_write":93,"num_write_kb":26702,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"262'313","reported_seq":1060,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-04-24T01:59:43.173067+0000","last_change":"2024-04-24T01:59:42.197591+0000","last_active":"2024-04-24T01:59:43.173067+0000","last_peered":"2024-04-24T01:59:43.173067+0000","last_clean":"2024-04-24T01:59:43.173067+0000","last_became_active":"2024-04-24T01:55:30.321866+0000","last_became_peered":"2024-04-24T01:55:30.321866+0000","last_unstale":"2024-04-24T01:59:43.173067+0000","last_undegraded":"2024-04-24T01:59:43.173067+0000","last_fullsized":"2024-04-24T01:59:43.173067+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":313,"log_dups_size":0,"ondisk_log_size":313,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":30,"snaptrim_duration":0,"stat_sum":{"num_bytes":5143140,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":165,"num_read_kb":113141,"num_write":151,"num_write_kb":40824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"264'229","reported_seq":992,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-04-24T01:59:43.190164+0000","last_change":"2024-04-24T01:59:42.169059+0000","last_active":"2024-04-24T01:59:43.190164+0000","last_peered":"2024-04-24T01:59:43.190164+0000","last_clean":"2024-04-24T01:59:43.190164+0000","last_became_active":"2024-04-24T01:55:30.320186+0000","last_became_peered":"2024-04-24T01:55:30.320186+0000","last_unstale":"2024-04-24T01:59:43.190164+0000","last_undegraded":"2024-04-24T01:59:43.190164+0000","last_fullsized":"2024-04-24T01:59:43.190164+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":229,"log_dups_size":0,"ondisk_log_size":229,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":24,"snaptrim_duration":0,"stat_sum":{"num_bytes":5159914,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":125,"num_read_kb":85296,"num_write":124,"num_write_kb":34018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"246'263","reported_seq":1041,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-04-24T01:59:44.071954+0000","last_change":"2024-04-24T01:59:44.086625+0000","last_active":"2024-04-24T01:59:44.071954+0000","last_peered":"2024-04-24T01:59:44.071954+0000","last_clean":"2024-04-24T01:59:44.071954+0000","last_became_active":"2024-04-24T01:55:30.288081+0000","last_became_peered":"2024-04-24T01:55:30.288081+0000","last_unstale":"2024-04-24T01:59:44.071954+0000","last_undegraded":"2024-04-24T01:59:44.071954+0000","last_fullsized":"2024-04-24T01:59:44.071954+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":263,"log_dups_size":0,"ondisk_log_size":263,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":27,"snaptrim_duration":0,"stat_sum":{"num_bytes":2269044,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":136,"num_read_kb":92553,"num_write":129,"num_write_kb":33953,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"263'101","reported_seq":872,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-04-24T01:59:43.106944+0000","last_change":"2024-04-24T01:59:42.205860+0000","last_active":"2024-04-24T01:59:43.106944+0000","last_peered":"2024-04-24T01:59:43.106944+0000","last_clean":"2024-04-24T01:59:43.106944+0000","last_became_active":"2024-04-24T01:55:30.333815+0000","last_became_peered":"2024-04-24T01:55:30.333815+0000","last_unstale":"2024-04-24T01:59:43.106944+0000","last_undegraded":"2024-04-24T01:59:43.106944+0000","last_fullsized":"2024-04-24T01:59:43.106944+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":10,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":45,"num_read_kb":23146,"num_write":56,"num_write_kb":15361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"267'539","reported_seq":1272,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-04-24T01:59:44.094844+0000","last_change":"2024-04-24T01:59:44.107994+0000","last_active":"2024-04-24T01:59:44.094844+0000","last_peered":"2024-04-24T01:59:44.094844+0000","last_clean":"2024-04-24T01:59:44.094844+0000","last_became_active":"2024-04-24T01:55:30.315075+0000","last_became_peered":"2024-04-24T01:55:30.315075+0000","last_unstale":"2024-04-24T01:59:44.094844+0000","last_undegraded":"2024-04-24T01:59:44.094844+0000","last_fullsized":"2024-04-24T01:59:44.094844+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":539,"log_dups_size":0,"ondisk_log_size":539,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":47,"snaptrim_duration":0,"stat_sum":{"num_bytes":17614330,"num_objects":11,"num_object_clones":5,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":245,"num_read_kb":153597,"num_write":250,"num_write_kb":65993,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"261'95","reported_seq":890,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-04-24T01:59:44.081793+0000","last_change":"2024-04-24T01:59:44.090162+0000","last_active":"2024-04-24T01:59:44.081793+0000","last_peered":"2024-04-24T01:59:44.081793+0000","last_clean":"2024-04-24T01:59:44.081793+0000","last_became_active":"2024-04-24T01:55:30.350300+0000","last_became_peered":"2024-04-24T01:55:30.350300+0000","last_unstale":"2024-04-24T01:59:44.081793+0000","last_undegraded":"2024-04-24T01:59:44.081793+0000","last_fullsized":"2024-04-24T01:59:44.081793+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":8,"snaptrim_duration":0,"stat_sum":{"num_bytes":3916097,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":46,"num_read_kb":22974,"num_write":62,"num_write_kb":15453,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"264'34","reported_seq":808,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-04-24T01:59:43.199536+0000","last_change":"2024-04-24T01:59:42.184597+0000","last_active":"2024-04-24T01:59:43.199536+0000","last_peered":"2024-04-24T01:59:43.199536+0000","last_clean":"2024-04-24T01:59:43.199536+0000","last_became_active":"2024-04-24T01:55:30.376813+0000","last_became_peered":"2024-04-24T01:55:30.376813+0000","last_unstale":"2024-04-24T01:59:43.199536+0000","last_undegraded":"2024-04-24T01:59:43.199536+0000","last_fullsized":"2024-04-24T01:59:43.199536+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0,"stat_sum":{"num_bytes":3025359,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":10082,"num_write":27,"num_write_kb":6802,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'97","reported_seq":802,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-04-24T01:59:43.101064+0000","last_change":"2024-04-24T01:55:16.196470+0000","last_active":"2024-04-24T01:59:43.101064+0000","last_peered":"2024-04-24T01:59:43.101064+0000","last_clean":"2024-04-24T01:59:43.101064+0000","last_became_active":"2024-04-24T01:55:16.190464+0000","last_became_peered":"2024-04-24T01:55:16.190464+0000","last_unstale":"2024-04-24T01:59:43.101064+0000","last_undegraded":"2024-04-24T01:59:43.101064+0000","last_fullsized":"2024-04-24T01:59:43.101064+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:15.127115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:15.127115+0000","last_clean_scrub_stamp":"2024-04-24T01:55:15.127115+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":663,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-04-24T01:59:43.198213+0000","last_change":"2024-04-24T01:55:18.236148+0000","last_active":"2024-04-24T01:59:43.198213+0000","last_peered":"2024-04-24T01:59:43.198213+0000","last_clean":"2024-04-24T01:59:43.198213+0000","last_became_active":"2024-04-24T01:55:18.230154+0000","last_became_peered":"2024-04-24T01:55:18.230154+0000","last_unstale":"2024-04-24T01:59:43.198213+0000","last_undegraded":"2024-04-24T01:59:43.198213+0000","last_fullsized":"2024-04-24T01:59:43.198213+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"261'385","reported_seq":1150,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-04-24T01:59:43.203002+0000","last_change":"2024-04-24T01:59:42.183207+0000","last_active":"2024-04-24T01:59:43.203002+0000","last_peered":"2024-04-24T01:59:43.203002+0000","last_clean":"2024-04-24T01:59:43.203002+0000","last_became_active":"2024-04-24T01:55:30.316222+0000","last_became_peered":"2024-04-24T01:55:30.316222+0000","last_unstale":"2024-04-24T01:59:43.203002+0000","last_undegraded":"2024-04-24T01:59:43.203002+0000","last_fullsized":"2024-04-24T01:59:43.203002+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":385,"log_dups_size":0,"ondisk_log_size":385,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":29,"snaptrim_duration":0,"stat_sum":{"num_bytes":12336701,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":212,"num_read_kb":149107,"num_write":205,"num_write_kb":57231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":684,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-04-24T01:59:44.084482+0000","last_change":"2024-04-24T01:55:18.265905+0000","last_active":"2024-04-24T01:59:44.084482+0000","last_peered":"2024-04-24T01:59:44.084482+0000","last_clean":"2024-04-24T01:59:44.084482+0000","last_became_active":"2024-04-24T01:55:18.257020+0000","last_became_peered":"2024-04-24T01:55:18.257020+0000","last_unstale":"2024-04-24T01:59:44.084482+0000","last_undegraded":"2024-04-24T01:59:44.084482+0000","last_fullsized":"2024-04-24T01:59:44.084482+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"260'317","reported_seq":1057,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-04-24T01:59:43.103050+0000","last_change":"2024-04-24T01:59:42.197776+0000","last_active":"2024-04-24T01:59:43.103050+0000","last_peered":"2024-04-24T01:59:43.103050+0000","last_clean":"2024-04-24T01:59:43.103050+0000","last_became_active":"2024-04-24T01:55:30.344129+0000","last_became_peered":"2024-04-24T01:55:30.344129+0000","last_unstale":"2024-04-24T01:59:43.103050+0000","last_undegraded":"2024-04-24T01:59:43.103050+0000","last_fullsized":"2024-04-24T01:59:43.103050+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":317,"log_dups_size":0,"ondisk_log_size":317,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":26,"snaptrim_duration":0,"stat_sum":{"num_bytes":3042166,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":133,"num_read_kb":72539,"num_write":149,"num_write_kb":41796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":663,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-04-24T01:59:43.187175+0000","last_change":"2024-04-24T01:55:18.215251+0000","last_active":"2024-04-24T01:59:43.187175+0000","last_peered":"2024-04-24T01:59:43.187175+0000","last_clean":"2024-04-24T01:59:43.187175+0000","last_became_active":"2024-04-24T01:55:18.206605+0000","last_became_peered":"2024-04-24T01:55:18.206605+0000","last_unstale":"2024-04-24T01:59:43.187175+0000","last_undegraded":"2024-04-24T01:59:43.187175+0000","last_fullsized":"2024-04-24T01:59:43.187175+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"247'144","reported_seq":938,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-04-24T01:59:43.109141+0000","last_change":"2024-04-24T01:59:42.207076+0000","last_active":"2024-04-24T01:59:43.109141+0000","last_peered":"2024-04-24T01:59:43.109141+0000","last_clean":"2024-04-24T01:59:43.109141+0000","last_became_active":"2024-04-24T01:55:30.401366+0000","last_became_peered":"2024-04-24T01:55:30.401366+0000","last_unstale":"2024-04-24T01:59:43.109141+0000","last_undegraded":"2024-04-24T01:59:43.109141+0000","last_fullsized":"2024-04-24T01:59:43.109141+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":144,"log_dups_size":0,"ondisk_log_size":144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":11,"snaptrim_duration":0,"stat_sum":{"num_bytes":5563282,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":130,"num_read_kb":104968,"num_write":55,"num_write_kb":14247,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":684,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-04-24T01:59:44.087577+0000","last_change":"2024-04-24T01:55:18.230893+0000","last_active":"2024-04-24T01:59:44.087577+0000","last_peered":"2024-04-24T01:59:44.087577+0000","last_clean":"2024-04-24T01:59:44.087577+0000","last_became_active":"2024-04-24T01:55:18.225188+0000","last_became_peered":"2024-04-24T01:55:18.225188+0000","last_unstale":"2024-04-24T01:59:44.087577+0000","last_undegraded":"2024-04-24T01:59:44.087577+0000","last_fullsized":"2024-04-24T01:59:44.087577+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"262'253","reported_seq":1001,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-04-24T01:59:43.168087+0000","last_change":"2024-04-24T01:59:42.199194+0000","last_active":"2024-04-24T01:59:43.168087+0000","last_peered":"2024-04-24T01:59:43.168087+0000","last_clean":"2024-04-24T01:59:43.168087+0000","last_became_active":"2024-04-24T01:55:30.343318+0000","last_became_peered":"2024-04-24T01:55:30.343318+0000","last_unstale":"2024-04-24T01:59:43.168087+0000","last_undegraded":"2024-04-24T01:59:43.168087+0000","last_fullsized":"2024-04-24T01:59:43.168087+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":253,"log_dups_size":0,"ondisk_log_size":253,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":23,"snaptrim_duration":0,"stat_sum":{"num_bytes":3092587,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":139,"num_read_kb":111498,"num_write":113,"num_write_kb":31479,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"264'334","reported_seq":1089,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-04-24T01:59:43.201953+0000","last_change":"2024-04-24T01:59:42.184291+0000","last_active":"2024-04-24T01:59:43.201953+0000","last_peered":"2024-04-24T01:59:43.201953+0000","last_clean":"2024-04-24T01:59:43.201953+0000","last_became_active":"2024-04-24T01:55:30.345308+0000","last_became_peered":"2024-04-24T01:55:30.345308+0000","last_unstale":"2024-04-24T01:59:43.201953+0000","last_undegraded":"2024-04-24T01:59:43.201953+0000","last_fullsized":"2024-04-24T01:59:43.201953+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":334,"log_dups_size":0,"ondisk_log_size":334,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":28,"snaptrim_duration":0,"stat_sum":{"num_bytes":8336569,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":178,"num_read_kb":114668,"num_write":162,"num_write_kb":42749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":655,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-04-24T01:59:43.180460+0000","last_change":"2024-04-24T01:55:18.224238+0000","last_active":"2024-04-24T01:59:43.180460+0000","last_peered":"2024-04-24T01:59:43.180460+0000","last_clean":"2024-04-24T01:59:43.180460+0000","last_became_active":"2024-04-24T01:55:18.219131+0000","last_became_peered":"2024-04-24T01:55:18.219131+0000","last_unstale":"2024-04-24T01:59:43.180460+0000","last_undegraded":"2024-04-24T01:59:43.180460+0000","last_fullsized":"2024-04-24T01:59:43.180460+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"265'70","reported_seq":839,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-04-24T01:59:43.177170+0000","last_change":"2024-04-24T01:59:42.241855+0000","last_active":"2024-04-24T01:59:43.177170+0000","last_peered":"2024-04-24T01:59:43.177170+0000","last_clean":"2024-04-24T01:59:43.177170+0000","last_became_active":"2024-04-24T01:55:30.319958+0000","last_became_peered":"2024-04-24T01:55:30.319958+0000","last_unstale":"2024-04-24T01:59:43.177170+0000","last_undegraded":"2024-04-24T01:59:43.177170+0000","last_fullsized":"2024-04-24T01:59:43.177170+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":5,"snaptrim_duration":0,"stat_sum":{"num_bytes":3126201,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":40,"num_read_kb":36473,"num_write":46,"num_write_kb":13159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":669,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-04-24T01:59:43.105277+0000","last_change":"2024-04-24T01:55:18.235763+0000","last_active":"2024-04-24T01:59:43.105277+0000","last_peered":"2024-04-24T01:59:43.105277+0000","last_clean":"2024-04-24T01:59:43.105277+0000","last_became_active":"2024-04-24T01:55:18.229175+0000","last_became_peered":"2024-04-24T01:55:18.229175+0000","last_unstale":"2024-04-24T01:59:43.105277+0000","last_undegraded":"2024-04-24T01:59:43.105277+0000","last_fullsized":"2024-04-24T01:59:43.105277+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"265'133","reported_seq":899,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-04-24T01:59:43.096197+0000","last_change":"2024-04-24T01:59:42.224330+0000","last_active":"2024-04-24T01:59:43.096197+0000","last_peered":"2024-04-24T01:59:43.096197+0000","last_clean":"2024-04-24T01:59:43.096197+0000","last_became_active":"2024-04-24T01:55:30.372690+0000","last_became_peered":"2024-04-24T01:55:30.372690+0000","last_unstale":"2024-04-24T01:59:43.096197+0000","last_undegraded":"2024-04-24T01:59:43.096197+0000","last_fullsized":"2024-04-24T01:59:43.096197+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":10,"snaptrim_duration":0,"stat_sum":{"num_bytes":3210236,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":61,"num_read_kb":39375,"num_write":64,"num_write_kb":18028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":655,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-04-24T01:59:43.176003+0000","last_change":"2024-04-24T01:55:18.278351+0000","last_active":"2024-04-24T01:59:43.176003+0000","last_peered":"2024-04-24T01:59:43.176003+0000","last_clean":"2024-04-24T01:59:43.176003+0000","last_became_active":"2024-04-24T01:55:18.273668+0000","last_became_peered":"2024-04-24T01:55:18.273668+0000","last_unstale":"2024-04-24T01:59:43.176003+0000","last_undegraded":"2024-04-24T01:59:43.176003+0000","last_fullsized":"2024-04-24T01:59:43.176003+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"264'262","reported_seq":1046,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-04-24T01:59:43.178433+0000","last_change":"2024-04-24T01:59:42.208500+0000","last_active":"2024-04-24T01:59:43.178433+0000","last_peered":"2024-04-24T01:59:43.178433+0000","last_clean":"2024-04-24T01:59:43.178433+0000","last_became_active":"2024-04-24T01:55:30.363123+0000","last_became_peered":"2024-04-24T01:55:30.363123+0000","last_unstale":"2024-04-24T01:59:43.178433+0000","last_undegraded":"2024-04-24T01:59:43.178433+0000","last_fullsized":"2024-04-24T01:59:43.178433+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":262,"log_dups_size":0,"ondisk_log_size":262,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":27,"snaptrim_duration":0,"stat_sum":{"num_bytes":12336635,"num_objects":7,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":195,"num_read_kb":157933,"num_write":147,"num_write_kb":39145,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":663,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-04-24T01:59:43.201186+0000","last_change":"2024-04-24T01:55:18.241117+0000","last_active":"2024-04-24T01:59:43.201186+0000","last_peered":"2024-04-24T01:59:43.201186+0000","last_clean":"2024-04-24T01:59:43.201186+0000","last_became_active":"2024-04-24T01:55:18.234313+0000","last_became_peered":"2024-04-24T01:55:18.234313+0000","last_unstale":"2024-04-24T01:59:43.201186+0000","last_undegraded":"2024-04-24T01:59:43.201186+0000","last_fullsized":"2024-04-24T01:59:43.201186+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":93634701,"num_objects":53,"num_object_clones":17,"num_object_copies":159,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":2013,"num_read_kb":1417249,"num_write":1833,"num_write_kb":496940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":213995520,"data_stored":234308265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3664,"ondisk_log_size":3664,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":1}],"osd_stats":[{"osd":3,"up_from":10,"seq":42949673229,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":89508,"kb_used_data":54364,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94282332,"statfs":{"total":96636764160,"available":96545107968,"internally_reserved":0,"allocated":55668736,"data_stored":59006824,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673229,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":84300,"kb_used_data":49156,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94287540,"statfs":{"total":96636764160,"available":96550440960,"internally_reserved":0,"allocated":50335744,"data_stored":54296994,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705934,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":86336,"kb_used_data":51192,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94285504,"statfs":{"total":96636764160,"available":96548356096,"internally_reserved":0,"allocated":52420608,"data_stored":57953580,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705933,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":96444,"kb_used_data":61300,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94275396,"statfs":{"total":96636764160,"available":96538005504,"internally_reserved":0,"allocated":62771200,"data_stored":66745965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":50176000,"data_stored":56411542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":62304256,"data_stored":67676981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":48091136,"data_stored":52754956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53424128,"data_stored":57464786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T01:59:46.645 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T01:59:46.751 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:59:46.749+0000 7f2660f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:59:46.755 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:59:46.754+0000 7f2660f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:59:47.077 INFO:tasks.rados.rados.0.smithi079.stdout:1129: done (1 left) 2024-04-24T01:59:47.078 INFO:tasks.rados.rados.0.smithi079.stdout:1130: write (excl) oid 23 current snap is 127 2024-04-24T01:59:47.078 INFO:tasks.rados.rados.0.smithi079.stdout:1127: finishing write tid 2 to smithi07929762-38 2024-04-24T01:59:47.078 INFO:tasks.rados.rados.0.smithi079.stdout:1130: seq_num 556 ranges {544692=732213,1985753=743863,3335184=9508} 2024-04-24T01:59:47.084 INFO:tasks.rados.rados.0.smithi079.stdout:1130: writing smithi07929762-23 from 544692 to 1276905 tid 1 2024-04-24T01:59:47.093 INFO:tasks.rados.rados.0.smithi079.stdout:1130: writing smithi07929762-23 from 1985753 to 2729616 tid 2 2024-04-24T01:59:47.097 INFO:tasks.rados.rados.0.smithi079.stdout:1130: writing smithi07929762-23 from 3335184 to 3344692 tid 3 2024-04-24T01:59:47.097 INFO:tasks.rados.rados.0.smithi079.stdout:1131: read oid 21 snap -1 2024-04-24T01:59:47.097 INFO:tasks.rados.rados.0.smithi079.stdout:1131: expect (ObjNum 500 snap 112 seq_num 500) 2024-04-24T01:59:47.097 INFO:tasks.rados.rados.0.smithi079.stdout:1132: delete oid 3 current snap is 127 2024-04-24T01:59:47.097 INFO:tasks.rados.rados.0.smithi079.stdout:1127: finishing write tid 3 to smithi07929762-38 2024-04-24T01:59:47.097 INFO:tasks.rados.rados.0.smithi079.stdout:1127: finishing write tid 4 to smithi07929762-38 2024-04-24T01:59:47.097 INFO:tasks.rados.rados.0.smithi079.stdout:1127: finishing write tid 5 to smithi07929762-38 2024-04-24T01:59:47.097 INFO:tasks.rados.rados.0.smithi079.stdout:1127: oid 38 updating version 0 to 267 2024-04-24T01:59:47.097 INFO:tasks.rados.rados.0.smithi079.stdout:1127: oid 38 version 267 is already newer than 267 2024-04-24T01:59:47.098 INFO:tasks.rados.rados.0.smithi079.stdout:1127: oid 38 version 267 is already newer than 267 2024-04-24T01:59:47.098 INFO:tasks.rados.rados.0.smithi079.stdout:1127: oid 38 version 267 is already newer than 267 2024-04-24T01:59:47.098 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 38 v 267 (ObjNum 554 snap 127 seq_num 554) dirty exists 2024-04-24T01:59:47.098 INFO:tasks.rados.rados.0.smithi079.stdout:1127: left oid 38 (ObjNum 554 snap 127 seq_num 554) 2024-04-24T01:59:47.166 INFO:tasks.rados.rados.0.smithi079.stdout:1131: expect (ObjNum 500 snap 112 seq_num 500) 2024-04-24T01:59:47.261 INFO:tasks.rados.rados.0.smithi079.stdout:1130: finishing write tid 1 to smithi07929762-23 2024-04-24T01:59:47.261 INFO:tasks.rados.rados.0.smithi079.stdout:1130: finishing write tid 2 to smithi07929762-23 2024-04-24T01:59:47.261 INFO:tasks.rados.rados.0.smithi079.stdout:1130: finishing write tid 3 to smithi07929762-23 2024-04-24T01:59:47.261 INFO:tasks.rados.rados.0.smithi079.stdout:1130: finishing write tid 4 to smithi07929762-23 2024-04-24T01:59:47.261 INFO:tasks.rados.rados.0.smithi079.stdout:1127: done (3 left) 2024-04-24T01:59:47.261 INFO:tasks.rados.rados.0.smithi079.stdout:1131: done (2 left) 2024-04-24T01:59:47.261 INFO:tasks.rados.rados.0.smithi079.stdout:1132: done (1 left) 2024-04-24T01:59:47.261 INFO:tasks.rados.rados.0.smithi079.stdout:1133: snap_create 2024-04-24T01:59:47.261 INFO:tasks.rados.rados.0.smithi079.stdout:1130: finishing write tid 5 to smithi07929762-23 2024-04-24T01:59:47.261 INFO:tasks.rados.rados.0.smithi079.stdout:1130: oid 23 updating version 0 to 99 2024-04-24T01:59:47.261 INFO:tasks.rados.rados.0.smithi079.stdout:1130: oid 23 version 99 is already newer than 99 2024-04-24T01:59:47.262 INFO:tasks.rados.rados.0.smithi079.stdout:1130: oid 23 version 99 is already newer than 96 2024-04-24T01:59:47.262 INFO:tasks.rados.rados.0.smithi079.stdout:1130: oid 23 version 99 is already newer than 99 2024-04-24T01:59:47.262 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 23 v 99 (ObjNum 556 snap 127 seq_num 556) dirty exists 2024-04-24T01:59:47.262 INFO:tasks.rados.rados.0.smithi079.stdout:1130: left oid 23 (ObjNum 556 snap 127 seq_num 556) 2024-04-24T01:59:48.085 INFO:tasks.rados.rados.0.smithi079.stdout:1130: done (1 left) 2024-04-24T01:59:48.085 INFO:tasks.rados.rados.0.smithi079.stdout:1133: done (0 left) 2024-04-24T01:59:48.085 INFO:tasks.rados.rados.0.smithi079.stdout:1134: write oid 36 current snap is 128 2024-04-24T01:59:48.085 INFO:tasks.rados.rados.0.smithi079.stdout:1134: seq_num 558 ranges {578306=482711,1586364=631179,2757324=484842} 2024-04-24T01:59:48.091 INFO:tasks.rados.rados.0.smithi079.stdout:1134: writing smithi07929762-36 from 578306 to 1061017 tid 1 2024-04-24T01:59:48.098 INFO:tasks.rados.rados.0.smithi079.stdout:1134: writing smithi07929762-36 from 1586364 to 2217543 tid 2 2024-04-24T01:59:48.106 INFO:tasks.rados.rados.0.smithi079.stdout:1134: writing smithi07929762-36 from 2757324 to 3242166 tid 3 2024-04-24T01:59:48.106 INFO:tasks.rados.rados.0.smithi079.stdout:1135: write oid 5 current snap is 128 2024-04-24T01:59:48.106 INFO:tasks.rados.rados.0.smithi079.stdout:1135: seq_num 559 ranges {595113=557960,1644846=491190,2784747=610366} 2024-04-24T01:59:48.115 INFO:tasks.rados.rados.0.smithi079.stdout:1135: writing smithi07929762-5 from 595113 to 1153073 tid 1 2024-04-24T01:59:48.120 INFO:tasks.rados.rados.0.smithi079.stdout:1135: writing smithi07929762-5 from 1644846 to 2136036 tid 2 2024-04-24T01:59:48.127 INFO:tasks.rados.rados.0.smithi079.stdout:1135: writing smithi07929762-5 from 2784747 to 3395113 tid 3 2024-04-24T01:59:48.127 INFO:tasks.rados.rados.0.smithi079.stdout:1136: snap_create 2024-04-24T01:59:48.168 INFO:tasks.rados.rados.0.smithi079.stdout:1135: finishing write tid 1 to smithi07929762-5 2024-04-24T01:59:48.188 INFO:tasks.rados.rados.0.smithi079.stdout:1135: finishing write tid 2 to smithi07929762-5 2024-04-24T01:59:48.210 INFO:tasks.rados.rados.0.smithi079.stdout:1135: finishing write tid 3 to smithi07929762-5 2024-04-24T01:59:48.210 INFO:tasks.rados.rados.0.smithi079.stdout:1134: finishing write tid 1 to smithi07929762-36 2024-04-24T01:59:48.213 INFO:tasks.rados.rados.0.smithi079.stdout:1135: finishing write tid 4 to smithi07929762-5 2024-04-24T01:59:48.215 INFO:tasks.rados.rados.0.smithi079.stdout:1135: finishing write tid 5 to smithi07929762-5 2024-04-24T01:59:48.215 INFO:tasks.rados.rados.0.smithi079.stdout:1135: oid 5 updating version 0 to 267 2024-04-24T01:59:48.215 INFO:tasks.rados.rados.0.smithi079.stdout:1135: oid 5 version 267 is already newer than 267 2024-04-24T01:59:48.215 INFO:tasks.rados.rados.0.smithi079.stdout:1135: oid 5 version 267 is already newer than 267 2024-04-24T01:59:48.215 INFO:tasks.rados.rados.0.smithi079.stdout:1135: oid 5 version 267 is already newer than 267 2024-04-24T01:59:48.215 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 5 v 267 (ObjNum 559 snap 128 seq_num 559) dirty exists 2024-04-24T01:59:48.215 INFO:tasks.rados.rados.0.smithi079.stdout:1135: left oid 5 (ObjNum 559 snap 128 seq_num 559) 2024-04-24T01:59:48.226 INFO:tasks.rados.rados.0.smithi079.stdout:1134: finishing write tid 2 to smithi07929762-36 2024-04-24T01:59:48.231 INFO:tasks.rados.rados.0.smithi079.stdout:1134: finishing write tid 3 to smithi07929762-36 2024-04-24T01:59:48.235 INFO:tasks.rados.rados.0.smithi079.stdout:1134: finishing write tid 4 to smithi07929762-36 2024-04-24T01:59:48.236 INFO:tasks.rados.rados.0.smithi079.stdout:1134: finishing write tid 5 to smithi07929762-36 2024-04-24T01:59:48.236 INFO:tasks.rados.rados.0.smithi079.stdout:1134: oid 36 updating version 0 to 105 2024-04-24T01:59:48.236 INFO:tasks.rados.rados.0.smithi079.stdout:1134: oid 36 version 105 is already newer than 105 2024-04-24T01:59:48.237 INFO:tasks.rados.rados.0.smithi079.stdout:1134: oid 36 version 105 is already newer than 105 2024-04-24T01:59:48.237 INFO:tasks.rados.rados.0.smithi079.stdout:1134: oid 36 version 105 is already newer than 105 2024-04-24T01:59:48.237 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 36 v 105 (ObjNum 558 snap 128 seq_num 558) dirty exists 2024-04-24T01:59:48.237 INFO:tasks.rados.rados.0.smithi079.stdout:1134: left oid 36 (ObjNum 558 snap 128 seq_num 558) 2024-04-24T01:59:48.848 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T01:59:48.954 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:59:48.952+0000 7f0275206640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:59:48.957 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:59:48.957+0000 7f0275206640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:59:49.086 INFO:tasks.rados.rados.0.smithi079.stdout:1134: done (2 left) 2024-04-24T01:59:49.087 INFO:tasks.rados.rados.0.smithi079.stdout:1135: done (1 left) 2024-04-24T01:59:49.087 INFO:tasks.rados.rados.0.smithi079.stdout:1136: done (0 left) 2024-04-24T01:59:49.087 INFO:tasks.rados.rados.0.smithi079.stdout:1137: write oid 34 current snap is 129 2024-04-24T01:59:49.087 INFO:tasks.rados.rados.0.smithi079.stdout:1137: seq_num 560 ranges {611920=633209,1703328=634848,2812170=507386} 2024-04-24T01:59:49.094 INFO:tasks.rados.rados.0.smithi079.stdout:1137: writing smithi07929762-34 from 611920 to 1245129 tid 1 2024-04-24T01:59:49.101 INFO:tasks.rados.rados.0.smithi079.stdout:1137: writing smithi07929762-34 from 1703328 to 2338176 tid 2 2024-04-24T01:59:49.109 INFO:tasks.rados.rados.0.smithi079.stdout:1137: writing smithi07929762-34 from 2812170 to 3319556 tid 3 2024-04-24T01:59:49.109 INFO:tasks.rados.rados.0.smithi079.stdout:1138: snap_create 2024-04-24T01:59:49.166 INFO:tasks.rados.rados.0.smithi079.stdout:1137: finishing write tid 1 to smithi07929762-34 2024-04-24T01:59:49.187 INFO:tasks.rados.rados.0.smithi079.stdout:1137: finishing write tid 2 to smithi07929762-34 2024-04-24T01:59:49.209 INFO:tasks.rados.rados.0.smithi079.stdout:1137: finishing write tid 3 to smithi07929762-34 2024-04-24T01:59:49.212 INFO:tasks.rados.rados.0.smithi079.stdout:1137: finishing write tid 4 to smithi07929762-34 2024-04-24T01:59:49.213 INFO:tasks.rados.rados.0.smithi079.stdout:1137: finishing write tid 5 to smithi07929762-34 2024-04-24T01:59:49.213 INFO:tasks.rados.rados.0.smithi079.stdout:1137: oid 34 updating version 0 to 272 2024-04-24T01:59:49.213 INFO:tasks.rados.rados.0.smithi079.stdout:1137: oid 34 version 272 is already newer than 272 2024-04-24T01:59:49.213 INFO:tasks.rados.rados.0.smithi079.stdout:1137: oid 34 version 272 is already newer than 272 2024-04-24T01:59:49.213 INFO:tasks.rados.rados.0.smithi079.stdout:1137: oid 34 version 272 is already newer than 272 2024-04-24T01:59:49.213 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 34 v 272 (ObjNum 560 snap 129 seq_num 560) dirty exists 2024-04-24T01:59:49.213 INFO:tasks.rados.rados.0.smithi079.stdout:1137: left oid 34 (ObjNum 560 snap 129 seq_num 560) 2024-04-24T01:59:50.090 INFO:tasks.rados.rados.0.smithi079.stdout:1137: done (1 left) 2024-04-24T01:59:50.091 INFO:tasks.rados.rados.0.smithi079.stdout:1138: done (0 left) 2024-04-24T01:59:50.091 INFO:tasks.rados.rados.0.smithi079.stdout:1139: snap_remove snap 124 2024-04-24T01:59:51.050 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T01:59:51.054 INFO:tasks.rados.rados.0.smithi079.stdout:1139: done (0 left) 2024-04-24T01:59:51.054 INFO:tasks.rados.rados.0.smithi079.stdout:1140: read oid 44 snap -1 2024-04-24T01:59:51.054 INFO:tasks.rados.rados.0.smithi079.stdout:1140: expect deleted 2024-04-24T01:59:51.054 INFO:tasks.rados.rados.0.smithi079.stdout:1141: write oid 12 current snap is 130 2024-04-24T01:59:51.054 INFO:tasks.rados.rados.0.smithi079.stdout:1141: seq_num 561 ranges {628727=708458,2045457=494859,3123240=305487} 2024-04-24T01:59:51.061 INFO:tasks.rados.rados.0.smithi079.stdout:1141: writing smithi07929762-12 from 628727 to 1337185 tid 1 2024-04-24T01:59:51.068 INFO:tasks.rados.rados.0.smithi079.stdout:1141: writing smithi07929762-12 from 2045457 to 2540316 tid 2 2024-04-24T01:59:51.073 INFO:tasks.rados.rados.0.smithi079.stdout:1141: writing smithi07929762-12 from 3123240 to 3428727 tid 3 2024-04-24T01:59:51.073 INFO:tasks.rados.rados.0.smithi079.stdout:1142: read oid 25 snap -1 2024-04-24T01:59:51.074 INFO:tasks.rados.rados.0.smithi079.stdout:1142: expect (ObjNum 508 snap 113 seq_num 508) 2024-04-24T01:59:51.074 INFO:tasks.rados.rados.0.smithi079.stdout:1143: read oid 36 snap -1 2024-04-24T01:59:51.074 INFO:tasks.rados.rados.0.smithi079.stdout:1143: expect (ObjNum 558 snap 128 seq_num 558) 2024-04-24T01:59:51.074 INFO:tasks.rados.rados.0.smithi079.stdout:1144: write oid 9 current snap is 130 2024-04-24T01:59:51.074 INFO:tasks.rados.rados.0.smithi079.stdout:1144: seq_num 562 ranges {645534=783707,2103939=638517,3150663=294871} 2024-04-24T01:59:51.081 INFO:tasks.rados.rados.0.smithi079.stdout:1144: writing smithi07929762-9 from 645534 to 1429241 tid 1 2024-04-24T01:59:51.089 INFO:tasks.rados.rados.0.smithi079.stdout:1144: writing smithi07929762-9 from 2103939 to 2742456 tid 2 2024-04-24T01:59:51.093 INFO:tasks.rados.rados.0.smithi079.stdout:1144: writing smithi07929762-9 from 3150663 to 3445534 tid 3 2024-04-24T01:59:51.093 INFO:tasks.rados.rados.0.smithi079.stdout:1145: read oid 22 snap 128 2024-04-24T01:59:51.093 INFO:tasks.rados.rados.0.smithi079.stdout:1145: expect deleted 2024-04-24T01:59:51.094 INFO:tasks.rados.rados.0.smithi079.stdout:1146: write (excl) oid 15 current snap is 130 2024-04-24T01:59:51.094 INFO:tasks.rados.rados.0.smithi079.stdout:1146: seq_num 563 ranges {662341=575309,1878774=498528,2894439=457555} 2024-04-24T01:59:51.100 INFO:tasks.rados.rados.0.smithi079.stdout:1146: writing smithi07929762-15 from 662341 to 1237650 tid 1 2024-04-24T01:59:51.107 INFO:tasks.rados.rados.0.smithi079.stdout:1146: writing smithi07929762-15 from 1878774 to 2377302 tid 2 2024-04-24T01:59:51.113 INFO:tasks.rados.rados.0.smithi079.stdout:1146: writing smithi07929762-15 from 2894439 to 3351994 tid 3 2024-04-24T01:59:51.113 INFO:tasks.rados.rados.0.smithi079.stdout:1147: read oid 48 snap -1 2024-04-24T01:59:51.113 INFO:tasks.rados.rados.0.smithi079.stdout:1147: expect (ObjNum 553 snap 127 seq_num 553) 2024-04-24T01:59:51.113 INFO:tasks.rados.rados.0.smithi079.stdout:1148: write oid 35 current snap is 130 2024-04-24T01:59:51.113 INFO:tasks.rados.rados.0.smithi079.stdout:1148: seq_num 564 ranges {679148=650558,1937256=642186,3321862=157286} 2024-04-24T01:59:51.120 INFO:tasks.rados.rados.0.smithi079.stdout:1148: writing smithi07929762-35 from 679148 to 1329706 tid 1 2024-04-24T01:59:51.127 INFO:tasks.rados.rados.0.smithi079.stdout:1148: writing smithi07929762-35 from 1937256 to 2579442 tid 2 2024-04-24T01:59:51.133 INFO:tasks.rados.rados.0.smithi079.stdout:1148: writing smithi07929762-35 from 3321862 to 3479148 tid 3 2024-04-24T01:59:51.133 INFO:tasks.rados.rados.0.smithi079.stdout:1149: write oid 20 current snap is 130 2024-04-24T01:59:51.133 INFO:tasks.rados.rados.0.smithi079.stdout:1149: seq_num 565 ranges {695955=725807,1879385=502197,2949285=480099} 2024-04-24T01:59:51.141 INFO:tasks.rados.rados.0.smithi079.stdout:1149: writing smithi07929762-20 from 695955 to 1421762 tid 1 2024-04-24T01:59:51.147 INFO:tasks.rados.rados.0.smithi079.stdout:1149: writing smithi07929762-20 from 1879385 to 2381582 tid 2 2024-04-24T01:59:51.155 INFO:tasks.rados.rados.0.smithi079.stdout:1149: writing smithi07929762-20 from 2949285 to 3429384 tid 3 2024-04-24T01:59:51.155 INFO:tasks.rados.rados.0.smithi079.stdout:1140: done (9 left) 2024-04-24T01:59:51.155 INFO:tasks.rados.rados.0.smithi079.stdout:1150: delete oid 30 current snap is 130 2024-04-24T01:59:51.186 INFO:tasks.rados.rados.0.smithi079.stdout:1141: finishing write tid 1 to smithi07929762-12 2024-04-24T01:59:51.206 INFO:tasks.rados.rados.0.smithi079.stdout:1143: expect (ObjNum 558 snap 128 seq_num 558) 2024-04-24T01:59:51.214 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:59:51.212+0000 7fb5e035d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:59:51.220 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:59:51.221+0000 7fb5e035d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:59:51.371 INFO:tasks.rados.rados.0.smithi079.stdout:1141: finishing write tid 2 to smithi07929762-12 2024-04-24T01:59:51.371 INFO:tasks.rados.rados.0.smithi079.stdout:1142: expect (ObjNum 508 snap 113 seq_num 508) 2024-04-24T01:59:51.534 INFO:tasks.rados.rados.0.smithi079.stdout:1141: finishing write tid 3 to smithi07929762-12 2024-04-24T01:59:51.534 INFO:tasks.rados.rados.0.smithi079.stdout:1141: finishing write tid 4 to smithi07929762-12 2024-04-24T01:59:51.535 INFO:tasks.rados.rados.0.smithi079.stdout:1141: finishing write tid 5 to smithi07929762-12 2024-04-24T01:59:51.535 INFO:tasks.rados.rados.0.smithi079.stdout:1141: oid 12 updating version 0 to 276 2024-04-24T01:59:51.535 INFO:tasks.rados.rados.0.smithi079.stdout:1141: oid 12 version 276 is already newer than 276 2024-04-24T01:59:51.535 INFO:tasks.rados.rados.0.smithi079.stdout:1141: oid 12 version 276 is already newer than 276 2024-04-24T01:59:51.535 INFO:tasks.rados.rados.0.smithi079.stdout:1141: oid 12 version 276 is already newer than 276 2024-04-24T01:59:51.535 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 12 v 276 (ObjNum 561 snap 130 seq_num 561) dirty exists 2024-04-24T01:59:51.535 INFO:tasks.rados.rados.0.smithi079.stdout:1141: left oid 12 (ObjNum 561 snap 130 seq_num 561) 2024-04-24T01:59:51.535 INFO:tasks.rados.rados.0.smithi079.stdout:1144: finishing write tid 1 to smithi07929762-9 2024-04-24T01:59:51.535 INFO:tasks.rados.rados.0.smithi079.stdout:1141: done (9 left) 2024-04-24T01:59:51.535 INFO:tasks.rados.rados.0.smithi079.stdout:1142: done (8 left) 2024-04-24T01:59:51.535 INFO:tasks.rados.rados.0.smithi079.stdout:1143: done (7 left) 2024-04-24T01:59:51.535 INFO:tasks.rados.rados.0.smithi079.stdout:1145: done (6 left) 2024-04-24T01:59:51.535 INFO:tasks.rados.rados.0.smithi079.stdout:1150: done (5 left) 2024-04-24T01:59:51.536 INFO:tasks.rados.rados.0.smithi079.stdout:1151: snap_remove snap 128 2024-04-24T01:59:51.622 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-04-24T01:59:51.623 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-04-24T01:59:51.623 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-04-24T01:59:51.623 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-04-24T01:59:51.731 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:59:51.727+0000 7f088d86d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:59:51.735 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:59:51.735+0000 7f088d86d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:59:51.749 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T01:59:51.749 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T01:59:51.749 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T01:59:51.749 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T01:59:51.749 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T01:59:51.749 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T01:59:51.749 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T01:59:51.749 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T01:59:51.750 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T01:59:51.750 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T01:59:51.750 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T01:59:51.750 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T01:59:51.752 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T01:59:51.752 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T01:59:51.753 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-04-24T01:59:51.859 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:59:51.854+0000 7fd10fead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:59:51.862 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:59:51.863+0000 7fd10fead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:59:51.877 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T01:59:51.877 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T01:59:51.877 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T01:59:51.877 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T01:59:51.877 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T01:59:51.877 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T01:59:51.877 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T01:59:51.877 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T01:59:51.877 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T01:59:51.877 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T01:59:51.877 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T01:59:51.878 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T01:59:51.880 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T01:59:51.880 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T01:59:51.880 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-04-24T01:59:51.987 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:59:51.983+0000 7f3ccfbff640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:59:51.991 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:59:51.991+0000 7f3cce99d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:59:52.008 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T01:59:52.008 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T01:59:52.008 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T01:59:52.008 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T01:59:52.008 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T01:59:52.008 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T01:59:52.008 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T01:59:52.008 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T01:59:52.008 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T01:59:52.008 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T01:59:52.008 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T01:59:52.009 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T01:59:52.010 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T01:59:52.011 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T01:59:52.011 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-04-24T01:59:52.107 INFO:tasks.rados.rados.0.smithi079.stdout:1151: done (5 left) 2024-04-24T01:59:52.107 INFO:tasks.rados.rados.0.smithi079.stdout:1152: read oid 42 snap 125 2024-04-24T01:59:52.107 INFO:tasks.rados.rados.0.smithi079.stdout:1152: expect deleted 2024-04-24T01:59:52.107 INFO:tasks.rados.rados.0.smithi079.stdout:1153: write oid 6 current snap is 130 2024-04-24T01:59:52.107 INFO:tasks.rados.rados.0.smithi079.stdout:1148: finishing write tid 1 to smithi07929762-35 2024-04-24T01:59:52.107 INFO:tasks.rados.rados.0.smithi079.stdout:1153: seq_num 567 ranges {729569=476305,1996349=505866,3004131=525438} 2024-04-24T01:59:52.114 INFO:tasks.rados.rados.0.smithi079.stdout:1153: writing smithi07929762-6 from 729569 to 1205874 tid 1 2024-04-24T01:59:52.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:59:52.115+0000 7f12df95d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:59:52.122 INFO:tasks.rados.rados.0.smithi079.stdout:1153: writing smithi07929762-6 from 1996349 to 2502215 tid 2 2024-04-24T01:59:52.125 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:59:52.126+0000 7f12df95d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:59:52.130 INFO:tasks.rados.rados.0.smithi079.stdout:1153: writing smithi07929762-6 from 3004131 to 3529569 tid 3 2024-04-24T01:59:52.131 INFO:tasks.rados.rados.0.smithi079.stdout:1154: read oid 16 snap -1 2024-04-24T01:59:52.131 INFO:tasks.rados.rados.0.smithi079.stdout:1154: expect (ObjNum 526 snap 120 seq_num 526) 2024-04-24T01:59:52.131 INFO:tasks.rados.rados.0.smithi079.stdout:1155: write (excl) oid 33 current snap is 130 2024-04-24T01:59:52.131 INFO:tasks.rados.rados.0.smithi079.stdout:1155: seq_num 568 ranges {746376=551554,2054831=649524,3315201=231175} 2024-04-24T01:59:52.140 INFO:tasks.rados.rados.0.smithi079.stdout:1155: writing smithi07929762-33 from 746376 to 1297930 tid 1 2024-04-24T01:59:52.148 INFO:tasks.rados.rados.0.smithi079.stdout:1155: writing smithi07929762-33 from 2054831 to 2704355 tid 2 2024-04-24T01:59:52.153 INFO:tasks.rados.rados.0.smithi079.stdout:1155: writing smithi07929762-33 from 3315201 to 3546376 tid 3 2024-04-24T01:59:52.153 INFO:tasks.rados.rados.0.smithi079.stdout:1156: read oid 1 snap -1 2024-04-24T01:59:52.153 INFO:tasks.rados.rados.0.smithi079.stdout:1156: expect (ObjNum 551 snap 126 seq_num 551) 2024-04-24T01:59:52.153 INFO:tasks.rados.rados.0.smithi079.stdout:1157: write oid 38 current snap is 130 2024-04-24T01:59:52.154 INFO:tasks.rados.rados.0.smithi079.stdout:1157: seq_num 569 ranges {763183=626803,1996960=793182,3226271=336912} 2024-04-24T01:59:52.158 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T01:59:52.158 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T01:59:52.158 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T01:59:52.158 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T01:59:52.158 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T01:59:52.158 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T01:59:52.159 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T01:59:52.159 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T01:59:52.159 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T01:59:52.159 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T01:59:52.159 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T01:59:52.159 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T01:59:52.163 INFO:tasks.rados.rados.0.smithi079.stdout:1157: writing smithi07929762-38 from 763183 to 1389986 tid 1 2024-04-24T01:59:52.164 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T01:59:52.164 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T01:59:52.173 INFO:tasks.rados.rados.0.smithi079.stdout:1157: writing smithi07929762-38 from 1996960 to 2790142 tid 2 2024-04-24T01:59:52.181 INFO:tasks.rados.rados.0.smithi079.stdout:1157: writing smithi07929762-38 from 3226271 to 3563183 tid 3 2024-04-24T01:59:52.181 INFO:tasks.rados.rados.0.smithi079.stdout:1158: read oid 25 snap 127 2024-04-24T01:59:52.181 INFO:tasks.rados.rados.0.smithi079.stdout:1158: expect (ObjNum 508 snap 113 seq_num 508) 2024-04-24T01:59:52.181 INFO:tasks.rados.rados.0.smithi079.stdout:1159: snap_remove snap 129 2024-04-24T01:59:53.109 INFO:tasks.rados.rados.0.smithi079.stdout:1159: done (12 left) 2024-04-24T01:59:53.109 INFO:tasks.rados.rados.0.smithi079.stdout:1160: write oid 13 current snap is 130 2024-04-24T01:59:53.109 INFO:tasks.rados.rados.0.smithi079.stdout:1147: expect (ObjNum 553 snap 127 seq_num 553) 2024-04-24T01:59:53.383 INFO:tasks.rados.rados.0.smithi079.stdout:1144: finishing write tid 2 to smithi07929762-9 2024-04-24T01:59:53.383 INFO:tasks.rados.rados.0.smithi079.stdout:1148: finishing write tid 2 to smithi07929762-35 2024-04-24T01:59:53.383 INFO:tasks.rados.rados.0.smithi079.stdout:1144: finishing write tid 3 to smithi07929762-9 2024-04-24T01:59:53.383 INFO:tasks.rados.rados.0.smithi079.stdout:1144: finishing write tid 4 to smithi07929762-9 2024-04-24T01:59:53.383 INFO:tasks.rados.rados.0.smithi079.stdout:1146: finishing write tid 1 to smithi07929762-15 2024-04-24T01:59:53.384 INFO:tasks.rados.rados.0.smithi079.stdout:1144: finishing write tid 5 to smithi07929762-9 2024-04-24T01:59:53.384 INFO:tasks.rados.rados.0.smithi079.stdout:1144: oid 9 updating version 0 to 339 2024-04-24T01:59:53.384 INFO:tasks.rados.rados.0.smithi079.stdout:1144: oid 9 version 339 is already newer than 339 2024-04-24T01:59:53.384 INFO:tasks.rados.rados.0.smithi079.stdout:1144: oid 9 version 339 is already newer than 339 2024-04-24T01:59:53.384 INFO:tasks.rados.rados.0.smithi079.stdout:1144: oid 9 version 339 is already newer than 339 2024-04-24T01:59:53.384 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 9 v 339 (ObjNum 562 snap 130 seq_num 562) dirty exists 2024-04-24T01:59:53.384 INFO:tasks.rados.rados.0.smithi079.stdout:1144: left oid 9 (ObjNum 562 snap 130 seq_num 562) 2024-04-24T01:59:53.384 INFO:tasks.rados.rados.0.smithi079.stdout:1148: finishing write tid 3 to smithi07929762-35 2024-04-24T01:59:53.384 INFO:tasks.rados.rados.0.smithi079.stdout:1160: seq_num 570 ranges {779990=702052,2055442=653193,3253694=326296} 2024-04-24T01:59:53.391 INFO:tasks.rados.rados.0.smithi079.stdout:1160: writing smithi07929762-13 from 779990 to 1482042 tid 1 2024-04-24T01:59:53.400 INFO:tasks.rados.rados.0.smithi079.stdout:1160: writing smithi07929762-13 from 2055442 to 2708635 tid 2 2024-04-24T01:59:53.407 INFO:tasks.rados.rados.0.smithi079.stdout:1160: writing smithi07929762-13 from 3253694 to 3579990 tid 3 2024-04-24T01:59:53.408 INFO:tasks.rados.rados.0.smithi079.stdout:1144: done (12 left) 2024-04-24T01:59:53.408 INFO:tasks.rados.rados.0.smithi079.stdout:1147: done (11 left) 2024-04-24T01:59:53.408 INFO:tasks.rados.rados.0.smithi079.stdout:1161: read oid 34 snap -1 2024-04-24T01:59:53.408 INFO:tasks.rados.rados.0.smithi079.stdout:1161: expect (ObjNum 560 snap 129 seq_num 560) 2024-04-24T01:59:53.408 INFO:tasks.rados.rados.0.smithi079.stdout:1162: delete oid 47 current snap is 130 2024-04-24T01:59:53.408 INFO:tasks.rados.rados.0.smithi079.stdout:1146: finishing write tid 2 to smithi07929762-15 2024-04-24T01:59:53.408 INFO:tasks.rados.rados.0.smithi079.stdout:1148: finishing write tid 4 to smithi07929762-35 2024-04-24T01:59:53.408 INFO:tasks.rados.rados.0.smithi079.stdout:1148: finishing write tid 5 to smithi07929762-35 2024-04-24T01:59:53.408 INFO:tasks.rados.rados.0.smithi079.stdout:1148: oid 35 updating version 0 to 322 2024-04-24T01:59:53.408 INFO:tasks.rados.rados.0.smithi079.stdout:1148: oid 35 version 322 is already newer than 322 2024-04-24T01:59:53.408 INFO:tasks.rados.rados.0.smithi079.stdout:1148: oid 35 version 322 is already newer than 322 2024-04-24T01:59:53.408 INFO:tasks.rados.rados.0.smithi079.stdout:1148: oid 35 version 322 is already newer than 322 2024-04-24T01:59:53.408 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 35 v 322 (ObjNum 564 snap 130 seq_num 564) dirty exists 2024-04-24T01:59:53.408 INFO:tasks.rados.rados.0.smithi079.stdout:1148: left oid 35 (ObjNum 564 snap 130 seq_num 564) 2024-04-24T01:59:53.408 INFO:tasks.rados.rados.0.smithi079.stdout:1146: finishing write tid 3 to smithi07929762-15 2024-04-24T01:59:53.409 INFO:tasks.rados.rados.0.smithi079.stdout:1146: finishing write tid 4 to smithi07929762-15 2024-04-24T01:59:53.409 INFO:tasks.rados.rados.0.smithi079.stdout:1149: finishing write tid 1 to smithi07929762-20 2024-04-24T01:59:53.409 INFO:tasks.rados.rados.0.smithi079.stdout:1146: finishing write tid 5 to smithi07929762-15 2024-04-24T01:59:53.409 INFO:tasks.rados.rados.0.smithi079.stdout:1146: oid 15 updating version 0 to 281 2024-04-24T01:59:53.409 INFO:tasks.rados.rados.0.smithi079.stdout:1146: oid 15 version 281 is already newer than 281 2024-04-24T01:59:53.409 INFO:tasks.rados.rados.0.smithi079.stdout:1146: oid 15 version 281 is already newer than 278 2024-04-24T01:59:53.409 INFO:tasks.rados.rados.0.smithi079.stdout:1146: oid 15 version 281 is already newer than 281 2024-04-24T01:59:53.409 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 15 v 281 (ObjNum 563 snap 130 seq_num 563) dirty exists 2024-04-24T01:59:53.409 INFO:tasks.rados.rados.0.smithi079.stdout:1146: left oid 15 (ObjNum 563 snap 130 seq_num 563) 2024-04-24T01:59:53.409 INFO:tasks.rados.rados.0.smithi079.stdout:1149: finishing write tid 2 to smithi07929762-20 2024-04-24T01:59:53.409 INFO:tasks.rados.rados.0.smithi079.stdout:1149: finishing write tid 3 to smithi07929762-20 2024-04-24T01:59:53.409 INFO:tasks.rados.rados.0.smithi079.stdout:1149: finishing write tid 4 to smithi07929762-20 2024-04-24T01:59:53.409 INFO:tasks.rados.rados.0.smithi079.stdout:1149: finishing write tid 5 to smithi07929762-20 2024-04-24T01:59:53.409 INFO:tasks.rados.rados.0.smithi079.stdout:1149: oid 20 updating version 0 to 344 2024-04-24T01:59:53.409 INFO:tasks.rados.rados.0.smithi079.stdout:1149: oid 20 version 344 is already newer than 344 2024-04-24T01:59:53.409 INFO:tasks.rados.rados.0.smithi079.stdout:1149: oid 20 version 344 is already newer than 344 2024-04-24T01:59:53.409 INFO:tasks.rados.rados.0.smithi079.stdout:1149: oid 20 version 344 is already newer than 344 2024-04-24T01:59:53.410 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 20 v 344 (ObjNum 565 snap 130 seq_num 565) dirty exists 2024-04-24T01:59:53.410 INFO:tasks.rados.rados.0.smithi079.stdout:1149: left oid 20 (ObjNum 565 snap 130 seq_num 565) 2024-04-24T01:59:53.410 INFO:tasks.rados.rados.0.smithi079.stdout:1158: expect (ObjNum 508 snap 113 seq_num 508) 2024-04-24T01:59:53.439 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T01:59:53.569 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:59:53.568+0000 7f90ea39d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:59:53.573 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:59:53.573+0000 7f90ea39d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:59:53.589 INFO:tasks.rados.rados.0.smithi079.stdout:1154: expect (ObjNum 526 snap 120 seq_num 526) 2024-04-24T01:59:53.775 INFO:tasks.rados.rados.0.smithi079.stdout:1155: finishing write tid 1 to smithi07929762-33 2024-04-24T01:59:53.775 INFO:tasks.rados.rados.0.smithi079.stdout:1156: expect (ObjNum 551 snap 126 seq_num 551) 2024-04-24T01:59:53.931 INFO:tasks.rados.rados.0.smithi079.stdout:1153: finishing write tid 1 to smithi07929762-6 2024-04-24T01:59:53.931 INFO:tasks.rados.rados.0.smithi079.stdout:1153: finishing write tid 2 to smithi07929762-6 2024-04-24T01:59:53.931 INFO:tasks.rados.rados.0.smithi079.stdout:1155: finishing write tid 2 to smithi07929762-33 2024-04-24T01:59:53.931 INFO:tasks.rados.rados.0.smithi079.stdout:1153: finishing write tid 3 to smithi07929762-6 2024-04-24T01:59:53.931 INFO:tasks.rados.rados.0.smithi079.stdout:1146: done (12 left) 2024-04-24T01:59:53.931 INFO:tasks.rados.rados.0.smithi079.stdout:1148: done (11 left) 2024-04-24T01:59:53.931 INFO:tasks.rados.rados.0.smithi079.stdout:1149: done (10 left) 2024-04-24T01:59:53.931 INFO:tasks.rados.rados.0.smithi079.stdout:1152: done (9 left) 2024-04-24T01:59:53.931 INFO:tasks.rados.rados.0.smithi079.stdout:1154: done (8 left) 2024-04-24T01:59:53.931 INFO:tasks.rados.rados.0.smithi079.stdout:1156: done (7 left) 2024-04-24T01:59:53.931 INFO:tasks.rados.rados.0.smithi079.stdout:1158: done (6 left) 2024-04-24T01:59:53.931 INFO:tasks.rados.rados.0.smithi079.stdout:1162: done (5 left) 2024-04-24T01:59:53.932 INFO:tasks.rados.rados.0.smithi079.stdout:1163: read oid 36 snap -1 2024-04-24T01:59:53.932 INFO:tasks.rados.rados.0.smithi079.stdout:1163: expect (ObjNum 558 snap 128 seq_num 558) 2024-04-24T01:59:53.932 INFO:tasks.rados.rados.0.smithi079.stdout:1164: snap_remove snap 117 2024-04-24T01:59:54.112 INFO:tasks.rados.rados.0.smithi079.stdout:1164: done (6 left) 2024-04-24T01:59:54.112 INFO:tasks.rados.rados.0.smithi079.stdout:1165: snap_create 2024-04-24T01:59:54.113 INFO:tasks.rados.rados.0.smithi079.stdout:1155: finishing write tid 3 to smithi07929762-33 2024-04-24T01:59:54.113 INFO:tasks.rados.rados.0.smithi079.stdout:1155: finishing write tid 4 to smithi07929762-33 2024-04-24T01:59:54.113 INFO:tasks.rados.rados.0.smithi079.stdout:1153: finishing write tid 4 to smithi07929762-6 2024-04-24T01:59:54.113 INFO:tasks.rados.rados.0.smithi079.stdout:1155: finishing write tid 5 to smithi07929762-33 2024-04-24T01:59:54.113 INFO:tasks.rados.rados.0.smithi079.stdout:1155: oid 33 updating version 0 to 352 2024-04-24T01:59:54.113 INFO:tasks.rados.rados.0.smithi079.stdout:1155: oid 33 version 352 is already newer than 352 2024-04-24T01:59:54.113 INFO:tasks.rados.rados.0.smithi079.stdout:1155: oid 33 version 352 is already newer than 352 2024-04-24T01:59:54.113 INFO:tasks.rados.rados.0.smithi079.stdout:1155: oid 33 version 352 is already newer than 349 2024-04-24T01:59:54.113 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 33 v 352 (ObjNum 568 snap 130 seq_num 568) dirty exists 2024-04-24T01:59:54.113 INFO:tasks.rados.rados.0.smithi079.stdout:1155: left oid 33 (ObjNum 568 snap 130 seq_num 568) 2024-04-24T01:59:54.113 INFO:tasks.rados.rados.0.smithi079.stdout:1153: finishing write tid 5 to smithi07929762-6 2024-04-24T01:59:54.113 INFO:tasks.rados.rados.0.smithi079.stdout:1153: oid 6 updating version 0 to 290 2024-04-24T01:59:54.114 INFO:tasks.rados.rados.0.smithi079.stdout:1153: oid 6 version 290 is already newer than 290 2024-04-24T01:59:54.114 INFO:tasks.rados.rados.0.smithi079.stdout:1153: oid 6 version 290 is already newer than 290 2024-04-24T01:59:54.114 INFO:tasks.rados.rados.0.smithi079.stdout:1153: oid 6 version 290 is already newer than 290 2024-04-24T01:59:54.114 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 6 v 290 (ObjNum 567 snap 130 seq_num 567) dirty exists 2024-04-24T01:59:54.114 INFO:tasks.rados.rados.0.smithi079.stdout:1153: left oid 6 (ObjNum 567 snap 130 seq_num 567) 2024-04-24T01:59:54.114 INFO:tasks.rados.rados.0.smithi079.stdout:1157: finishing write tid 1 to smithi07929762-38 2024-04-24T01:59:54.114 INFO:tasks.rados.rados.0.smithi079.stdout:1157: finishing write tid 2 to smithi07929762-38 2024-04-24T01:59:54.114 INFO:tasks.rados.rados.0.smithi079.stdout:1157: finishing write tid 3 to smithi07929762-38 2024-04-24T01:59:54.114 INFO:tasks.rados.rados.0.smithi079.stdout:1157: finishing write tid 4 to smithi07929762-38 2024-04-24T01:59:54.114 INFO:tasks.rados.rados.0.smithi079.stdout:1157: finishing write tid 5 to smithi07929762-38 2024-04-24T01:59:54.114 INFO:tasks.rados.rados.0.smithi079.stdout:1157: oid 38 updating version 0 to 295 2024-04-24T01:59:54.114 INFO:tasks.rados.rados.0.smithi079.stdout:1157: oid 38 version 295 is already newer than 295 2024-04-24T01:59:54.114 INFO:tasks.rados.rados.0.smithi079.stdout:1157: oid 38 version 295 is already newer than 295 2024-04-24T01:59:54.115 INFO:tasks.rados.rados.0.smithi079.stdout:1157: oid 38 version 295 is already newer than 295 2024-04-24T01:59:54.115 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 38 v 295 (ObjNum 569 snap 130 seq_num 569) dirty exists 2024-04-24T01:59:54.115 INFO:tasks.rados.rados.0.smithi079.stdout:1157: left oid 38 (ObjNum 569 snap 130 seq_num 569) 2024-04-24T01:59:54.115 INFO:tasks.rados.rados.0.smithi079.stdout:1160: finishing write tid 1 to smithi07929762-13 2024-04-24T01:59:54.115 INFO:tasks.rados.rados.0.smithi079.stdout:1160: finishing write tid 2 to smithi07929762-13 2024-04-24T01:59:54.115 INFO:tasks.rados.rados.0.smithi079.stdout:1160: finishing write tid 3 to smithi07929762-13 2024-04-24T01:59:54.115 INFO:tasks.rados.rados.0.smithi079.stdout:1160: finishing write tid 4 to smithi07929762-13 2024-04-24T01:59:54.115 INFO:tasks.rados.rados.0.smithi079.stdout:1160: finishing write tid 5 to smithi07929762-13 2024-04-24T01:59:54.115 INFO:tasks.rados.rados.0.smithi079.stdout:1160: oid 13 updating version 0 to 390 2024-04-24T01:59:54.115 INFO:tasks.rados.rados.0.smithi079.stdout:1160: oid 13 version 390 is already newer than 390 2024-04-24T01:59:54.115 INFO:tasks.rados.rados.0.smithi079.stdout:1160: oid 13 version 390 is already newer than 389 2024-04-24T01:59:54.115 INFO:tasks.rados.rados.0.smithi079.stdout:1160: oid 13 version 390 is already newer than 390 2024-04-24T01:59:54.115 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 13 v 390 (ObjNum 570 snap 130 seq_num 570) dirty exists 2024-04-24T01:59:54.115 INFO:tasks.rados.rados.0.smithi079.stdout:1160: left oid 13 (ObjNum 570 snap 130 seq_num 570) 2024-04-24T01:59:54.116 INFO:tasks.rados.rados.0.smithi079.stdout:1161: expect (ObjNum 560 snap 129 seq_num 560) 2024-04-24T01:59:54.421 INFO:tasks.rados.rados.0.smithi079.stdout:1163: expect (ObjNum 558 snap 128 seq_num 558) 2024-04-24T01:59:55.110 INFO:tasks.rados.rados.0.smithi079.stdout:1153: done (6 left) 2024-04-24T01:59:55.111 INFO:tasks.rados.rados.0.smithi079.stdout:1155: done (5 left) 2024-04-24T01:59:55.111 INFO:tasks.rados.rados.0.smithi079.stdout:1157: done (4 left) 2024-04-24T01:59:55.111 INFO:tasks.rados.rados.0.smithi079.stdout:1160: done (3 left) 2024-04-24T01:59:55.111 INFO:tasks.rados.rados.0.smithi079.stdout:1161: done (2 left) 2024-04-24T01:59:55.111 INFO:tasks.rados.rados.0.smithi079.stdout:1163: done (1 left) 2024-04-24T01:59:55.111 INFO:tasks.rados.rados.0.smithi079.stdout:1165: done (0 left) 2024-04-24T01:59:55.111 INFO:tasks.rados.rados.0.smithi079.stdout:1166: write oid 35 current snap is 131 2024-04-24T01:59:55.111 INFO:tasks.rados.rados.0.smithi079.stdout:1166: seq_num 572 ranges {413604=568903,1488759=656862,2624893=591709} 2024-04-24T01:59:55.116 INFO:tasks.rados.rados.0.smithi079.stdout:1166: writing smithi07929762-35 from 413604 to 982507 tid 1 2024-04-24T01:59:55.124 INFO:tasks.rados.rados.0.smithi079.stdout:1166: writing smithi07929762-35 from 1488759 to 2145621 tid 2 2024-04-24T01:59:55.131 INFO:tasks.rados.rados.0.smithi079.stdout:1166: writing smithi07929762-35 from 2624893 to 3216602 tid 3 2024-04-24T01:59:55.131 INFO:tasks.rados.rados.0.smithi079.stdout:1167: read oid 33 snap 125 2024-04-24T01:59:55.131 INFO:tasks.rados.rados.0.smithi079.stdout:1167: expect deleted 2024-04-24T01:59:55.131 INFO:tasks.rados.rados.0.smithi079.stdout:1168: delete oid 26 current snap is 131 2024-04-24T01:59:55.234 INFO:tasks.rados.rados.0.smithi079.stdout:1166: finishing write tid 1 to smithi07929762-35 2024-04-24T01:59:55.239 INFO:tasks.rados.rados.0.smithi079.stdout:1166: finishing write tid 2 to smithi07929762-35 2024-04-24T01:59:55.240 INFO:tasks.rados.rados.0.smithi079.stdout:1167: done (2 left) 2024-04-24T01:59:55.240 INFO:tasks.rados.rados.0.smithi079.stdout:1168: done (1 left) 2024-04-24T01:59:55.240 INFO:tasks.rados.rados.0.smithi079.stdout:1169: read oid 34 snap -1 2024-04-24T01:59:55.240 INFO:tasks.rados.rados.0.smithi079.stdout:1169: expect (ObjNum 560 snap 129 seq_num 560) 2024-04-24T01:59:55.240 INFO:tasks.rados.rados.0.smithi079.stdout:1170: write oid 30 current snap is 131 2024-04-24T01:59:55.240 INFO:tasks.rados.rados.0.smithi079.stdout:1170: seq_num 574 ranges {447218=719401,1889370=660531,2963386=614253} 2024-04-24T01:59:55.246 INFO:tasks.rados.rados.0.smithi079.stdout:1170: writing smithi07929762-30 from 447218 to 1166619 tid 1 2024-04-24T01:59:55.254 INFO:tasks.rados.rados.0.smithi079.stdout:1170: writing smithi07929762-30 from 1889370 to 2549901 tid 2 2024-04-24T01:59:55.261 INFO:tasks.rados.rados.0.smithi079.stdout:1170: writing smithi07929762-30 from 2963386 to 3577639 tid 3 2024-04-24T01:59:55.261 INFO:tasks.rados.rados.0.smithi079.stdout:1171: write (excl) oid 28 current snap is 131 2024-04-24T01:59:55.261 INFO:tasks.rados.rados.0.smithi079.stdout:1166: finishing write tid 3 to smithi07929762-35 2024-04-24T01:59:55.261 INFO:tasks.rados.rados.0.smithi079.stdout:1171: seq_num 575 ranges {464025=794650,1947852=404189,2990809=425525} 2024-04-24T01:59:55.270 INFO:tasks.rados.rados.0.smithi079.stdout:1171: writing smithi07929762-28 from 464025 to 1258675 tid 1 2024-04-24T01:59:55.276 INFO:tasks.rados.rados.0.smithi079.stdout:1171: writing smithi07929762-28 from 1947852 to 2352041 tid 2 2024-04-24T01:59:55.283 INFO:tasks.rados.rados.0.smithi079.stdout:1171: writing smithi07929762-28 from 2990809 to 3416334 tid 3 2024-04-24T01:59:55.284 INFO:tasks.rados.rados.0.smithi079.stdout:1172: snap_remove snap 130 2024-04-24T01:59:55.677 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T01:59:55.782 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:59:55.781+0000 7faf46aed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:59:55.786 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:59:55.786+0000 7faf46aed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:59:56.133 INFO:tasks.rados.rados.0.smithi079.stdout:1172: done (4 left) 2024-04-24T01:59:56.133 INFO:tasks.rados.rados.0.smithi079.stdout:1173: delete oid 47 current snap is 131 2024-04-24T01:59:56.133 INFO:tasks.rados.rados.0.smithi079.stdout:1166: finishing write tid 4 to smithi07929762-35 2024-04-24T01:59:56.133 INFO:tasks.rados.rados.0.smithi079.stdout:1166: finishing write tid 5 to smithi07929762-35 2024-04-24T01:59:56.133 INFO:tasks.rados.rados.0.smithi079.stdout:1166: oid 35 updating version 0 to 331 2024-04-24T01:59:56.133 INFO:tasks.rados.rados.0.smithi079.stdout:1166: oid 35 version 331 is already newer than 331 2024-04-24T01:59:56.133 INFO:tasks.rados.rados.0.smithi079.stdout:1166: oid 35 version 331 is already newer than 331 2024-04-24T01:59:56.133 INFO:tasks.rados.rados.0.smithi079.stdout:1166: oid 35 version 331 is already newer than 331 2024-04-24T01:59:56.133 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 35 v 331 (ObjNum 572 snap 131 seq_num 572) dirty exists 2024-04-24T01:59:56.133 INFO:tasks.rados.rados.0.smithi079.stdout:1166: left oid 35 (ObjNum 572 snap 131 seq_num 572) 2024-04-24T01:59:56.133 INFO:tasks.rados.rados.0.smithi079.stdout:1169: expect (ObjNum 560 snap 129 seq_num 560) 2024-04-24T01:59:56.399 INFO:tasks.rados.rados.0.smithi079.stdout:1170: finishing write tid 1 to smithi07929762-30 2024-04-24T01:59:56.400 INFO:tasks.rados.rados.0.smithi079.stdout:1170: finishing write tid 2 to smithi07929762-30 2024-04-24T01:59:56.400 INFO:tasks.rados.rados.0.smithi079.stdout:1170: finishing write tid 3 to smithi07929762-30 2024-04-24T01:59:56.400 INFO:tasks.rados.rados.0.smithi079.stdout:1170: finishing write tid 4 to smithi07929762-30 2024-04-24T01:59:56.400 INFO:tasks.rados.rados.0.smithi079.stdout:1171: finishing write tid 1 to smithi07929762-28 2024-04-24T01:59:56.400 INFO:tasks.rados.rados.0.smithi079.stdout:1170: finishing write tid 5 to smithi07929762-30 2024-04-24T01:59:56.400 INFO:tasks.rados.rados.0.smithi079.stdout:1170: oid 30 updating version 0 to 145 2024-04-24T01:59:56.400 INFO:tasks.rados.rados.0.smithi079.stdout:1170: oid 30 updating version 145 to 147 2024-04-24T01:59:56.400 INFO:tasks.rados.rados.0.smithi079.stdout:1170: oid 30 version 147 is already newer than 147 2024-04-24T01:59:56.400 INFO:tasks.rados.rados.0.smithi079.stdout:1170: oid 30 version 147 is already newer than 147 2024-04-24T01:59:56.400 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 30 v 147 (ObjNum 574 snap 131 seq_num 574) dirty exists 2024-04-24T01:59:56.400 INFO:tasks.rados.rados.0.smithi079.stdout:1170: left oid 30 (ObjNum 574 snap 131 seq_num 574) 2024-04-24T01:59:56.400 INFO:tasks.rados.rados.0.smithi079.stdout:1171: finishing write tid 2 to smithi07929762-28 2024-04-24T01:59:56.400 INFO:tasks.rados.rados.0.smithi079.stdout:1171: finishing write tid 3 to smithi07929762-28 2024-04-24T01:59:56.400 INFO:tasks.rados.rados.0.smithi079.stdout:1171: finishing write tid 4 to smithi07929762-28 2024-04-24T01:59:56.400 INFO:tasks.rados.rados.0.smithi079.stdout:1171: finishing write tid 5 to smithi07929762-28 2024-04-24T01:59:56.400 INFO:tasks.rados.rados.0.smithi079.stdout:1171: oid 28 updating version 0 to 317 2024-04-24T01:59:56.401 INFO:tasks.rados.rados.0.smithi079.stdout:1171: oid 28 version 317 is already newer than 317 2024-04-24T01:59:56.401 INFO:tasks.rados.rados.0.smithi079.stdout:1171: oid 28 version 317 is already newer than 317 2024-04-24T01:59:56.401 INFO:tasks.rados.rados.0.smithi079.stdout:1171: oid 28 version 317 is already newer than 314 2024-04-24T01:59:56.401 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 28 v 317 (ObjNum 575 snap 131 seq_num 575) dirty exists 2024-04-24T01:59:56.401 INFO:tasks.rados.rados.0.smithi079.stdout:1171: left oid 28 (ObjNum 575 snap 131 seq_num 575) 2024-04-24T01:59:56.401 INFO:tasks.rados.rados.0.smithi079.stdout:1166: done (4 left) 2024-04-24T01:59:56.401 INFO:tasks.rados.rados.0.smithi079.stdout:1169: done (3 left) 2024-04-24T01:59:56.401 INFO:tasks.rados.rados.0.smithi079.stdout:1170: done (2 left) 2024-04-24T01:59:56.401 INFO:tasks.rados.rados.0.smithi079.stdout:1171: done (1 left) 2024-04-24T01:59:56.401 INFO:tasks.rados.rados.0.smithi079.stdout:1173: done (0 left) 2024-04-24T01:59:56.401 INFO:tasks.rados.rados.0.smithi079.stdout:1174: snap_create 2024-04-24T01:59:57.135 INFO:tasks.rados.rados.0.smithi079.stdout:1174: done (0 left) 2024-04-24T01:59:57.136 INFO:tasks.rados.rados.0.smithi079.stdout:1175: write (excl) oid 9 current snap is 132 2024-04-24T01:59:57.136 INFO:tasks.rados.rados.0.smithi079.stdout:1175: seq_num 577 ranges {497639=545148,1548463=407858,2529302=564422,3528143=169496} 2024-04-24T01:59:57.141 INFO:tasks.rados.rados.0.smithi079.stdout:1175: writing smithi07929762-9 from 497639 to 1042787 tid 1 2024-04-24T01:59:57.147 INFO:tasks.rados.rados.0.smithi079.stdout:1175: writing smithi07929762-9 from 1548463 to 1956321 tid 2 2024-04-24T01:59:57.154 INFO:tasks.rados.rados.0.smithi079.stdout:1175: writing smithi07929762-9 from 2529302 to 3093724 tid 3 2024-04-24T01:59:57.157 INFO:tasks.rados.rados.0.smithi079.stdout:1175: writing smithi07929762-9 from 3528143 to 3697639 tid 4 2024-04-24T01:59:57.158 INFO:tasks.rados.rados.0.smithi079.stdout:1176: delete oid 39 current snap is 132 2024-04-24T01:59:57.165 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-04-24T01:59:57.165 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-04-24T01:59:57.165 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T01:59:57.221 INFO:tasks.rados.rados.0.smithi079.stdout:1176: done (1 left) 2024-04-24T01:59:57.221 INFO:tasks.rados.rados.0.smithi079.stdout:1177: read oid 19 snap -1 2024-04-24T01:59:57.222 INFO:tasks.rados.rados.0.smithi079.stdout:1177: expect deleted 2024-04-24T01:59:57.222 INFO:tasks.rados.rados.0.smithi079.stdout:1178: snap_remove snap 125 2024-04-24T01:59:57.289 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:59:57.285+0000 7fd6521ff640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:59:57.293 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:59:57.293+0000 7fd6521ff640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:59:57.480 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T01:59:57.480 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-24T01:59:57.493 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":413,"stamp":"2024-04-24T01:59:57.133807+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":106460895,"num_objects":65,"num_object_clones":25,"num_object_copies":195,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":2187,"num_read_kb":1481248,"num_write":2042,"num_write_kb":520600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3879,"ondisk_log_size":3879,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":422948,"kb_used_data":282372,"kb_used_omap":0,"kb_used_meta":140544,"kb_avail":377064412,"statfs":{"total":386547056640,"available":386113957888,"internally_reserved":0,"allocated":289148928,"data_stored":323958691,"data_compressed":89889,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":143917056},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":7193594,"num_objects":4,"num_object_clones":4,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":-1,"num_read":59,"num_read_kb":60207,"num_write":40,"num_write_kb":12329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.026450"},"pg_stats":[{"pgid":"3.f","version":"274'203","reported_seq":1042,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-04-24T01:59:53.455759+0000","last_change":"2024-04-24T01:59:53.465770+0000","last_active":"2024-04-24T01:59:53.455759+0000","last_peered":"2024-04-24T01:59:53.455759+0000","last_clean":"2024-04-24T01:59:53.455759+0000","last_became_active":"2024-04-24T01:55:30.386726+0000","last_became_peered":"2024-04-24T01:55:30.386726+0000","last_unstale":"2024-04-24T01:59:53.455759+0000","last_undegraded":"2024-04-24T01:59:53.455759+0000","last_fullsized":"2024-04-24T01:59:53.455759+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":203,"log_dups_size":0,"ondisk_log_size":203,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":18,"snaptrim_duration":0,"stat_sum":{"num_bytes":5580089,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":155,"num_read_kb":137682,"num_write":98,"num_write_kb":28444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"262'313","reported_seq":1099,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-24T01:59:54.140715+0000","last_change":"2024-04-24T01:59:54.147763+0000","last_active":"2024-04-24T01:59:54.140715+0000","last_peered":"2024-04-24T01:59:54.140715+0000","last_clean":"2024-04-24T01:59:54.140715+0000","last_became_active":"2024-04-24T01:55:30.321866+0000","last_became_peered":"2024-04-24T01:55:30.321866+0000","last_unstale":"2024-04-24T01:59:54.140715+0000","last_undegraded":"2024-04-24T01:59:54.140715+0000","last_fullsized":"2024-04-24T01:59:54.140715+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":313,"log_dups_size":0,"ondisk_log_size":313,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":30,"snaptrim_duration":0,"stat_sum":{"num_bytes":5143140,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":171,"num_read_kb":116837,"num_write":151,"num_write_kb":40824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"264'229","reported_seq":1022,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-04-24T01:59:53.118806+0000","last_change":"2024-04-24T01:59:53.129142+0000","last_active":"2024-04-24T01:59:53.118806+0000","last_peered":"2024-04-24T01:59:53.118806+0000","last_clean":"2024-04-24T01:59:53.118806+0000","last_became_active":"2024-04-24T01:55:30.320186+0000","last_became_peered":"2024-04-24T01:55:30.320186+0000","last_unstale":"2024-04-24T01:59:53.118806+0000","last_undegraded":"2024-04-24T01:59:53.118806+0000","last_fullsized":"2024-04-24T01:59:53.118806+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":229,"log_dups_size":0,"ondisk_log_size":229,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":24,"snaptrim_duration":0,"stat_sum":{"num_bytes":5159914,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":125,"num_read_kb":85296,"num_write":124,"num_write_kb":34018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"275'276","reported_seq":1088,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-04-24T01:59:53.520099+0000","last_change":"2024-04-24T01:59:53.466111+0000","last_active":"2024-04-24T01:59:53.520099+0000","last_peered":"2024-04-24T01:59:53.520099+0000","last_clean":"2024-04-24T01:59:53.520099+0000","last_became_active":"2024-04-24T01:55:30.288081+0000","last_became_peered":"2024-04-24T01:55:30.288081+0000","last_unstale":"2024-04-24T01:59:53.520099+0000","last_undegraded":"2024-04-24T01:59:53.520099+0000","last_fullsized":"2024-04-24T01:59:53.520099+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":276,"log_dups_size":0,"ondisk_log_size":276,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":27,"snaptrim_duration":0,"stat_sum":{"num_bytes":6807033,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":144,"num_read_kb":101516,"num_write":139,"num_write_kb":37310,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"271'105","reported_seq":911,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-04-24T01:59:53.988059+0000","last_change":"2024-04-24T01:59:53.977969+0000","last_active":"2024-04-24T01:59:53.988059+0000","last_peered":"2024-04-24T01:59:53.988059+0000","last_clean":"2024-04-24T01:59:53.988059+0000","last_became_active":"2024-04-24T01:55:30.333815+0000","last_became_peered":"2024-04-24T01:55:30.333815+0000","last_unstale":"2024-04-24T01:59:53.988059+0000","last_undegraded":"2024-04-24T01:59:53.988059+0000","last_fullsized":"2024-04-24T01:59:53.988059+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":10,"snaptrim_duration":0,"stat_sum":{"num_bytes":3378306,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":58,"num_read_kb":37733,"num_write":61,"num_write_kb":16924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"274'549","reported_seq":1315,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-04-24T01:59:53.452129+0000","last_change":"2024-04-24T01:59:53.464860+0000","last_active":"2024-04-24T01:59:53.452129+0000","last_peered":"2024-04-24T01:59:53.452129+0000","last_clean":"2024-04-24T01:59:53.452129+0000","last_became_active":"2024-04-24T01:55:30.315075+0000","last_became_peered":"2024-04-24T01:55:30.315075+0000","last_unstale":"2024-04-24T01:59:53.452129+0000","last_undegraded":"2024-04-24T01:59:53.452129+0000","last_fullsized":"2024-04-24T01:59:53.452129+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":549,"log_dups_size":0,"ondisk_log_size":549,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":47,"snaptrim_duration":0,"stat_sum":{"num_bytes":14387287,"num_objects":12,"num_object_clones":6,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":257,"num_read_kb":168346,"num_write":251,"num_write_kb":65993,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"270'99","reported_seq":927,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-04-24T01:59:53.462700+0000","last_change":"2024-04-24T01:59:53.472060+0000","last_active":"2024-04-24T01:59:53.462700+0000","last_peered":"2024-04-24T01:59:53.462700+0000","last_clean":"2024-04-24T01:59:53.462700+0000","last_became_active":"2024-04-24T01:55:30.350300+0000","last_became_peered":"2024-04-24T01:55:30.350300+0000","last_unstale":"2024-04-24T01:59:53.462700+0000","last_undegraded":"2024-04-24T01:59:53.462700+0000","last_fullsized":"2024-04-24T01:59:53.462700+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":8,"snaptrim_duration":0,"stat_sum":{"num_bytes":7260789,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":48,"num_read_kb":22975,"num_write":67,"num_write_kb":16906,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"264'34","reported_seq":838,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-04-24T01:59:53.130005+0000","last_change":"2024-04-24T01:59:53.141164+0000","last_active":"2024-04-24T01:59:53.130005+0000","last_peered":"2024-04-24T01:59:53.130005+0000","last_clean":"2024-04-24T01:59:53.130005+0000","last_became_active":"2024-04-24T01:55:30.376813+0000","last_became_peered":"2024-04-24T01:55:30.376813+0000","last_unstale":"2024-04-24T01:59:53.130005+0000","last_undegraded":"2024-04-24T01:59:53.130005+0000","last_fullsized":"2024-04-24T01:59:53.130005+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0,"stat_sum":{"num_bytes":3025359,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":10082,"num_write":27,"num_write_kb":6802,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'97","reported_seq":824,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-04-24T01:59:53.958131+0000","last_change":"2024-04-24T01:55:16.196470+0000","last_active":"2024-04-24T01:59:53.958131+0000","last_peered":"2024-04-24T01:59:53.958131+0000","last_clean":"2024-04-24T01:59:53.958131+0000","last_became_active":"2024-04-24T01:55:16.190464+0000","last_became_peered":"2024-04-24T01:55:16.190464+0000","last_unstale":"2024-04-24T01:59:53.958131+0000","last_undegraded":"2024-04-24T01:59:53.958131+0000","last_fullsized":"2024-04-24T01:59:53.958131+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:15.127115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:15.127115+0000","last_clean_scrub_stamp":"2024-04-24T01:55:15.127115+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":687,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-04-24T01:59:53.131881+0000","last_change":"2024-04-24T01:55:18.236148+0000","last_active":"2024-04-24T01:59:53.131881+0000","last_peered":"2024-04-24T01:59:53.131881+0000","last_clean":"2024-04-24T01:59:53.131881+0000","last_became_active":"2024-04-24T01:55:18.230154+0000","last_became_peered":"2024-04-24T01:55:18.230154+0000","last_unstale":"2024-04-24T01:59:53.131881+0000","last_undegraded":"2024-04-24T01:59:53.131881+0000","last_fullsized":"2024-04-24T01:59:53.131881+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"261'385","reported_seq":1186,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-04-24T01:59:53.136399+0000","last_change":"2024-04-24T01:59:53.139985+0000","last_active":"2024-04-24T01:59:53.136399+0000","last_peered":"2024-04-24T01:59:53.136399+0000","last_clean":"2024-04-24T01:59:53.136399+0000","last_became_active":"2024-04-24T01:55:30.316222+0000","last_became_peered":"2024-04-24T01:55:30.316222+0000","last_unstale":"2024-04-24T01:59:53.136399+0000","last_undegraded":"2024-04-24T01:59:53.136399+0000","last_fullsized":"2024-04-24T01:59:53.136399+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":385,"log_dups_size":0,"ondisk_log_size":385,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":29,"snaptrim_duration":0,"stat_sum":{"num_bytes":12336701,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":224,"num_read_kb":163227,"num_write":205,"num_write_kb":57231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":711,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-04-24T01:59:53.468538+0000","last_change":"2024-04-24T01:55:18.265905+0000","last_active":"2024-04-24T01:59:53.468538+0000","last_peered":"2024-04-24T01:59:53.468538+0000","last_clean":"2024-04-24T01:59:53.468538+0000","last_became_active":"2024-04-24T01:55:18.257020+0000","last_became_peered":"2024-04-24T01:55:18.257020+0000","last_unstale":"2024-04-24T01:59:53.468538+0000","last_undegraded":"2024-04-24T01:59:53.468538+0000","last_fullsized":"2024-04-24T01:59:53.468538+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"274'322","reported_seq":1089,"reported_epoch":276,"state":"active+clean+snaptrim","last_fresh":"2024-04-24T01:59:53.953812+0000","last_change":"2024-04-24T01:59:53.953533+0000","last_active":"2024-04-24T01:59:53.953812+0000","last_peered":"2024-04-24T01:59:53.953812+0000","last_clean":"2024-04-24T01:59:53.953812+0000","last_became_active":"2024-04-24T01:55:30.344129+0000","last_became_peered":"2024-04-24T01:55:30.344129+0000","last_unstale":"2024-04-24T01:59:53.953812+0000","last_undegraded":"2024-04-24T01:59:53.953812+0000","last_fullsized":"2024-04-24T01:59:53.953812+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":322,"log_dups_size":0,"ondisk_log_size":322,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":26,"snaptrim_duration":0,"stat_sum":{"num_bytes":3479148,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":134,"num_read_kb":72540,"num_write":154,"num_write_kb":43214,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":687,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-04-24T01:59:53.121345+0000","last_change":"2024-04-24T01:55:18.215251+0000","last_active":"2024-04-24T01:59:53.121345+0000","last_peered":"2024-04-24T01:59:53.121345+0000","last_clean":"2024-04-24T01:59:53.121345+0000","last_became_active":"2024-04-24T01:55:18.206605+0000","last_became_peered":"2024-04-24T01:55:18.206605+0000","last_unstale":"2024-04-24T01:59:53.121345+0000","last_undegraded":"2024-04-24T01:59:53.121345+0000","last_fullsized":"2024-04-24T01:59:53.121345+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"274'150","reported_seq":969,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-04-24T01:59:53.972733+0000","last_change":"2024-04-24T01:59:53.974732+0000","last_active":"2024-04-24T01:59:53.972733+0000","last_peered":"2024-04-24T01:59:53.972733+0000","last_clean":"2024-04-24T01:59:53.972733+0000","last_became_active":"2024-04-24T01:55:30.401366+0000","last_became_peered":"2024-04-24T01:55:30.401366+0000","last_unstale":"2024-04-24T01:59:53.972733+0000","last_undegraded":"2024-04-24T01:59:53.972733+0000","last_fullsized":"2024-04-24T01:59:53.972733+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":150,"log_dups_size":0,"ondisk_log_size":150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":11,"snaptrim_duration":0,"stat_sum":{"num_bytes":2252237,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":130,"num_read_kb":104968,"num_write":56,"num_write_kb":14247,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":711,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-04-24T01:59:53.466402+0000","last_change":"2024-04-24T01:55:18.230893+0000","last_active":"2024-04-24T01:59:53.466402+0000","last_peered":"2024-04-24T01:59:53.466402+0000","last_clean":"2024-04-24T01:59:53.466402+0000","last_became_active":"2024-04-24T01:55:18.225188+0000","last_became_peered":"2024-04-24T01:55:18.225188+0000","last_unstale":"2024-04-24T01:59:53.466402+0000","last_undegraded":"2024-04-24T01:59:53.466402+0000","last_fullsized":"2024-04-24T01:59:53.466402+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"262'253","reported_seq":1036,"reported_epoch":277,"state":"active+clean+snaptrim","last_fresh":"2024-04-24T01:59:54.129357+0000","last_change":"2024-04-24T01:59:54.128979+0000","last_active":"2024-04-24T01:59:54.129357+0000","last_peered":"2024-04-24T01:59:54.129357+0000","last_clean":"2024-04-24T01:59:54.129357+0000","last_became_active":"2024-04-24T01:55:30.343318+0000","last_became_peered":"2024-04-24T01:55:30.343318+0000","last_unstale":"2024-04-24T01:59:54.129357+0000","last_undegraded":"2024-04-24T01:59:54.129357+0000","last_fullsized":"2024-04-24T01:59:54.129357+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":253,"log_dups_size":0,"ondisk_log_size":253,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":24,"snaptrim_duration":0,"stat_sum":{"num_bytes":3092587,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":139,"num_read_kb":111498,"num_write":113,"num_write_kb":31479,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"276'356","reported_seq":1138,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-04-24T01:59:53.148611+0000","last_change":"2024-04-24T01:59:53.161867+0000","last_active":"2024-04-24T01:59:53.148611+0000","last_peered":"2024-04-24T01:59:53.148611+0000","last_clean":"2024-04-24T01:59:53.148611+0000","last_became_active":"2024-04-24T01:55:30.345308+0000","last_became_peered":"2024-04-24T01:55:30.345308+0000","last_unstale":"2024-04-24T01:59:53.148611+0000","last_undegraded":"2024-04-24T01:59:53.148611+0000","last_fullsized":"2024-04-24T01:59:53.148611+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":356,"log_dups_size":0,"ondisk_log_size":356,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":28,"snaptrim_duration":0,"stat_sum":{"num_bytes":13647680,"num_objects":6,"num_object_clones":2,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":182,"num_read_kb":114671,"num_write":177,"num_write_kb":47496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":684,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-24T01:59:54.140142+0000","last_change":"2024-04-24T01:55:18.224238+0000","last_active":"2024-04-24T01:59:54.140142+0000","last_peered":"2024-04-24T01:59:54.140142+0000","last_clean":"2024-04-24T01:59:54.140142+0000","last_became_active":"2024-04-24T01:55:18.219131+0000","last_became_peered":"2024-04-24T01:55:18.219131+0000","last_unstale":"2024-04-24T01:59:54.140142+0000","last_undegraded":"2024-04-24T01:59:54.140142+0000","last_fullsized":"2024-04-24T01:59:54.140142+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"265'70","reported_seq":874,"reported_epoch":277,"state":"active+clean+snaptrim","last_fresh":"2024-04-24T01:59:54.141800+0000","last_change":"2024-04-24T01:59:54.141346+0000","last_active":"2024-04-24T01:59:54.141800+0000","last_peered":"2024-04-24T01:59:54.141800+0000","last_clean":"2024-04-24T01:59:54.141800+0000","last_became_active":"2024-04-24T01:55:30.319958+0000","last_became_peered":"2024-04-24T01:55:30.319958+0000","last_unstale":"2024-04-24T01:59:54.141800+0000","last_undegraded":"2024-04-24T01:59:54.141800+0000","last_fullsized":"2024-04-24T01:59:54.141800+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0,"stat_sum":{"num_bytes":3126201,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":40,"num_read_kb":36473,"num_write":46,"num_write_kb":13159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":691,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-04-24T01:59:53.967493+0000","last_change":"2024-04-24T01:55:18.235763+0000","last_active":"2024-04-24T01:59:53.967493+0000","last_peered":"2024-04-24T01:59:53.967493+0000","last_clean":"2024-04-24T01:59:53.967493+0000","last_became_active":"2024-04-24T01:55:18.229175+0000","last_became_peered":"2024-04-24T01:55:18.229175+0000","last_unstale":"2024-04-24T01:59:53.967493+0000","last_undegraded":"2024-04-24T01:59:53.967493+0000","last_fullsized":"2024-04-24T01:59:53.967493+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"274'141","reported_seq":931,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-04-24T01:59:53.966258+0000","last_change":"2024-04-24T01:59:53.974637+0000","last_active":"2024-04-24T01:59:53.966258+0000","last_peered":"2024-04-24T01:59:53.966258+0000","last_clean":"2024-04-24T01:59:53.966258+0000","last_became_active":"2024-04-24T01:55:30.372690+0000","last_became_peered":"2024-04-24T01:55:30.372690+0000","last_unstale":"2024-04-24T01:59:53.966258+0000","last_undegraded":"2024-04-24T01:59:53.966258+0000","last_fullsized":"2024-04-24T01:59:53.966258+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":11,"snaptrim_duration":0,"stat_sum":{"num_bytes":3210236,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":61,"num_read_kb":39375,"num_write":64,"num_write_kb":18028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":684,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-24T01:59:54.144161+0000","last_change":"2024-04-24T01:55:18.278351+0000","last_active":"2024-04-24T01:59:54.144161+0000","last_peered":"2024-04-24T01:59:54.144161+0000","last_clean":"2024-04-24T01:59:54.144161+0000","last_became_active":"2024-04-24T01:55:18.273668+0000","last_became_peered":"2024-04-24T01:55:18.273668+0000","last_unstale":"2024-04-24T01:59:54.144161+0000","last_undegraded":"2024-04-24T01:59:54.144161+0000","last_fullsized":"2024-04-24T01:59:54.144161+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"277'297","reported_seq":1111,"reported_epoch":277,"state":"active+clean+snaptrim","last_fresh":"2024-04-24T01:59:54.159779+0000","last_change":"2024-04-24T01:59:54.142477+0000","last_active":"2024-04-24T01:59:54.159779+0000","last_peered":"2024-04-24T01:59:54.159779+0000","last_clean":"2024-04-24T01:59:54.159779+0000","last_became_active":"2024-04-24T01:55:30.363123+0000","last_became_peered":"2024-04-24T01:55:30.363123+0000","last_unstale":"2024-04-24T01:59:54.159779+0000","last_undegraded":"2024-04-24T01:59:54.159779+0000","last_fullsized":"2024-04-24T01:59:54.159779+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":297,"log_dups_size":0,"ondisk_log_size":297,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":29,"snaptrim_duration":0,"stat_sum":{"num_bytes":13983820,"num_objects":10,"num_object_clones":6,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":201,"num_read_kb":157938,"num_write":172,"num_write_kb":46631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":687,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-04-24T01:59:53.133555+0000","last_change":"2024-04-24T01:55:18.241117+0000","last_active":"2024-04-24T01:59:53.133555+0000","last_peered":"2024-04-24T01:59:53.133555+0000","last_clean":"2024-04-24T01:59:53.133555+0000","last_became_active":"2024-04-24T01:55:18.234313+0000","last_became_peered":"2024-04-24T01:55:18.234313+0000","last_unstale":"2024-04-24T01:59:53.133555+0000","last_undegraded":"2024-04-24T01:59:53.133555+0000","last_fullsized":"2024-04-24T01:59:53.133555+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":105870527,"num_objects":63,"num_object_clones":25,"num_object_copies":189,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":2084,"num_read_kb":1481157,"num_write":1905,"num_write_kb":518706,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":278188032,"data_stored":312787591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3782,"ondisk_log_size":3782,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":1}],"osd_stats":[{"osd":3,"up_from":10,"seq":42949673239,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":109576,"kb_used_data":74432,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94262264,"statfs":{"total":96636764160,"available":96524558336,"internally_reserved":0,"allocated":76218368,"data_stored":85531977,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673239,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":103108,"kb_used_data":67964,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94268732,"statfs":{"total":96636764160,"available":96531181568,"internally_reserved":0,"allocated":69595136,"data_stored":78774266,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705944,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":95552,"kb_used_data":60408,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94276288,"statfs":{"total":96636764160,"available":96538918912,"internally_reserved":0,"allocated":61857792,"data_stored":68618989,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705943,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114712,"kb_used_data":79568,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94257128,"statfs":{"total":96636764160,"available":96519299072,"internally_reserved":0,"allocated":81477632,"data_stored":91033459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":59518976,"data_stored":65921938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":79220736,"data_stored":90048776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65560576,"data_stored":73978242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73887744,"data_stored":82838635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T01:59:57.879 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T01:59:57.984 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:59:57.983+0000 7f8c8a26d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:59:57.988 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T01:59:57.988+0000 7f8c8a26d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T01:59:58.136 INFO:tasks.rados.rados.0.smithi079.stdout:1178: done (2 left) 2024-04-24T01:59:58.136 INFO:tasks.rados.rados.0.smithi079.stdout:1179: snap_create 2024-04-24T01:59:58.136 INFO:tasks.rados.rados.0.smithi079.stdout:1175: finishing write tid 1 to smithi07929762-9 2024-04-24T01:59:58.136 INFO:tasks.rados.rados.0.smithi079.stdout:1175: finishing write tid 2 to smithi07929762-9 2024-04-24T01:59:58.136 INFO:tasks.rados.rados.0.smithi079.stdout:1175: finishing write tid 3 to smithi07929762-9 2024-04-24T01:59:58.136 INFO:tasks.rados.rados.0.smithi079.stdout:1175: finishing write tid 4 to smithi07929762-9 2024-04-24T01:59:58.136 INFO:tasks.rados.rados.0.smithi079.stdout:1175: finishing write tid 5 to smithi07929762-9 2024-04-24T01:59:58.136 INFO:tasks.rados.rados.0.smithi079.stdout:1175: finishing write tid 6 to smithi07929762-9 2024-04-24T01:59:58.136 INFO:tasks.rados.rados.0.smithi079.stdout:1175: oid 9 updating version 0 to 360 2024-04-24T01:59:58.136 INFO:tasks.rados.rados.0.smithi079.stdout:1175: oid 9 updating version 360 to 364 2024-04-24T01:59:58.137 INFO:tasks.rados.rados.0.smithi079.stdout:1175: oid 9 version 364 is already newer than 364 2024-04-24T01:59:58.137 INFO:tasks.rados.rados.0.smithi079.stdout:1175: oid 9 version 364 is already newer than 364 2024-04-24T01:59:58.137 INFO:tasks.rados.rados.0.smithi079.stdout:1175: oid 9 version 364 is already newer than 364 2024-04-24T01:59:58.137 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 9 v 364 (ObjNum 577 snap 132 seq_num 577) dirty exists 2024-04-24T01:59:58.137 INFO:tasks.rados.rados.0.smithi079.stdout:1175: left oid 9 (ObjNum 577 snap 132 seq_num 577) 2024-04-24T01:59:59.139 INFO:tasks.rados.rados.0.smithi079.stdout:1175: done (2 left) 2024-04-24T01:59:59.139 INFO:tasks.rados.rados.0.smithi079.stdout:1177: done (1 left) 2024-04-24T01:59:59.140 INFO:tasks.rados.rados.0.smithi079.stdout:1179: done (0 left) 2024-04-24T01:59:59.140 INFO:tasks.rados.rados.0.smithi079.stdout:1180: snap_create 2024-04-24T02:00:00.083 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T02:00:00.140 INFO:tasks.rados.rados.0.smithi079.stdout:1180: done (0 left) 2024-04-24T02:00:00.140 INFO:tasks.rados.rados.0.smithi079.stdout:1181: write oid 23 current snap is 134 2024-04-24T02:00:00.140 INFO:tasks.rados.rados.0.smithi079.stdout:1181: seq_num 579 ranges {531253=411999,1381780=411527,2300501=586966,3376974=354279} 2024-04-24T02:00:00.147 INFO:tasks.rados.rados.0.smithi079.stdout:1181: writing smithi07929762-23 from 531253 to 943252 tid 1 2024-04-24T02:00:00.152 INFO:tasks.rados.rados.0.smithi079.stdout:1181: writing smithi07929762-23 from 1381780 to 1793307 tid 2 2024-04-24T02:00:00.160 INFO:tasks.rados.rados.0.smithi079.stdout:1181: writing smithi07929762-23 from 2300501 to 2887467 tid 3 2024-04-24T02:00:00.165 INFO:tasks.rados.rados.0.smithi079.stdout:1181: writing smithi07929762-23 from 3376974 to 3731253 tid 4 2024-04-24T02:00:00.165 INFO:tasks.rados.rados.0.smithi079.stdout:1182: snap_create 2024-04-24T02:00:00.240 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:00.236+0000 7fa39142d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:00.245 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:00.245+0000 7fa39142d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:00.263 INFO:tasks.rados.rados.0.smithi079.stdout:1181: finishing write tid 1 to smithi07929762-23 2024-04-24T02:00:00.271 INFO:tasks.rados.rados.0.smithi079.stdout:1181: finishing write tid 2 to smithi07929762-23 2024-04-24T02:00:00.286 INFO:tasks.rados.rados.0.smithi079.stdout:1181: finishing write tid 3 to smithi07929762-23 2024-04-24T02:00:00.294 INFO:tasks.rados.rados.0.smithi079.stdout:1181: finishing write tid 4 to smithi07929762-23 2024-04-24T02:00:00.296 INFO:tasks.rados.rados.0.smithi079.stdout:1181: finishing write tid 5 to smithi07929762-23 2024-04-24T02:00:00.298 INFO:tasks.rados.rados.0.smithi079.stdout:1181: finishing write tid 6 to smithi07929762-23 2024-04-24T02:00:00.298 INFO:tasks.rados.rados.0.smithi079.stdout:1181: oid 23 updating version 0 to 107 2024-04-24T02:00:00.298 INFO:tasks.rados.rados.0.smithi079.stdout:1181: oid 23 version 107 is already newer than 107 2024-04-24T02:00:00.298 INFO:tasks.rados.rados.0.smithi079.stdout:1181: oid 23 version 107 is already newer than 107 2024-04-24T02:00:00.298 INFO:tasks.rados.rados.0.smithi079.stdout:1181: oid 23 version 107 is already newer than 107 2024-04-24T02:00:00.298 INFO:tasks.rados.rados.0.smithi079.stdout:1181: oid 23 version 107 is already newer than 107 2024-04-24T02:00:00.298 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 23 v 107 (ObjNum 579 snap 134 seq_num 579) dirty exists 2024-04-24T02:00:00.298 INFO:tasks.rados.rados.0.smithi079.stdout:1181: left oid 23 (ObjNum 579 snap 134 seq_num 579) 2024-04-24T02:00:01.057 INFO:tasks.rados.rados.0.smithi079.stdout:1181: done (1 left) 2024-04-24T02:00:01.058 INFO:tasks.rados.rados.0.smithi079.stdout:1182: done (0 left) 2024-04-24T02:00:01.058 INFO:tasks.rados.rados.0.smithi079.stdout:1183: snap_remove snap 133 2024-04-24T02:00:02.057 INFO:tasks.rados.rados.0.smithi079.stdout:1183: done (0 left) 2024-04-24T02:00:02.057 INFO:tasks.rados.rados.0.smithi079.stdout:1184: read oid 15 snap 132 2024-04-24T02:00:02.057 INFO:tasks.rados.rados.0.smithi079.stdout:1184: expect (ObjNum 563 snap 130 seq_num 563) 2024-04-24T02:00:02.057 INFO:tasks.rados.rados.0.smithi079.stdout:1185: read oid 4 snap 132 2024-04-24T02:00:02.057 INFO:tasks.rados.rados.0.smithi079.stdout:1185: expect (ObjNum 471 snap 107 seq_num 471) 2024-04-24T02:00:02.058 INFO:tasks.rados.rados.0.smithi079.stdout:1186: write oid 39 current snap is 135 2024-04-24T02:00:02.058 INFO:tasks.rados.rados.0.smithi079.stdout:1186: seq_num 580 ranges {548060=487248,1440262=671538,2844277=798238} 2024-04-24T02:00:02.063 INFO:tasks.rados.rados.0.smithi079.stdout:1186: writing smithi07929762-39 from 548060 to 1035308 tid 1 2024-04-24T02:00:02.069 INFO:tasks.rados.rados.0.smithi079.stdout:1186: writing smithi07929762-39 from 1440262 to 2111800 tid 2 2024-04-24T02:00:02.080 INFO:tasks.rados.rados.0.smithi079.stdout:1186: writing smithi07929762-39 from 2844277 to 3642515 tid 3 2024-04-24T02:00:02.080 INFO:tasks.rados.rados.0.smithi079.stdout:1187: snap_create 2024-04-24T02:00:02.119 INFO:tasks.rados.rados.0.smithi079.stdout:1185: expect (ObjNum 471 snap 107 seq_num 471) 2024-04-24T02:00:02.324 INFO:tasks.rados.rados.0.smithi079.stdout:1186: finishing write tid 1 to smithi07929762-39 2024-04-24T02:00:02.324 INFO:tasks.rados.rados.0.smithi079.stdout:1186: finishing write tid 2 to smithi07929762-39 2024-04-24T02:00:02.324 INFO:tasks.rados.rados.0.smithi079.stdout:1186: finishing write tid 3 to smithi07929762-39 2024-04-24T02:00:02.324 INFO:tasks.rados.rados.0.smithi079.stdout:1184: expect (ObjNum 563 snap 130 seq_num 563) 2024-04-24T02:00:02.363 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T02:00:02.479 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:02.477+0000 7faaea064640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:02.483 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:02.482+0000 7faaea064640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:02.495 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-04-24T02:00:02.495 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-04-24T02:00:02.495 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-04-24T02:00:02.496 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-04-24T02:00:02.526 INFO:tasks.rados.rados.0.smithi079.stdout:1186: finishing write tid 4 to smithi07929762-39 2024-04-24T02:00:02.526 INFO:tasks.rados.rados.0.smithi079.stdout:1186: finishing write tid 5 to smithi07929762-39 2024-04-24T02:00:02.526 INFO:tasks.rados.rados.0.smithi079.stdout:1186: oid 39 updating version 0 to 260 2024-04-24T02:00:02.526 INFO:tasks.rados.rados.0.smithi079.stdout:1186: oid 39 version 260 is already newer than 260 2024-04-24T02:00:02.526 INFO:tasks.rados.rados.0.smithi079.stdout:1186: oid 39 version 260 is already newer than 260 2024-04-24T02:00:02.526 INFO:tasks.rados.rados.0.smithi079.stdout:1186: oid 39 version 260 is already newer than 260 2024-04-24T02:00:02.526 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 39 v 260 (ObjNum 580 snap 135 seq_num 580) dirty exists 2024-04-24T02:00:02.526 INFO:tasks.rados.rados.0.smithi079.stdout:1186: left oid 39 (ObjNum 580 snap 135 seq_num 580) 2024-04-24T02:00:02.606 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:02.600+0000 7fee3f79d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:02.610 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:02.609+0000 7fee3f79d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:02.624 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T02:00:02.625 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T02:00:02.625 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T02:00:02.625 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T02:00:02.625 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T02:00:02.625 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T02:00:02.625 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T02:00:02.625 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T02:00:02.625 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T02:00:02.625 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T02:00:02.625 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T02:00:02.625 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T02:00:02.627 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T02:00:02.628 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T02:00:02.628 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-04-24T02:00:02.777 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:02.771+0000 7fd49d26d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:02.781 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:02.780+0000 7fd49d26d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:02.795 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T02:00:02.795 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T02:00:02.795 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T02:00:02.795 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T02:00:02.795 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T02:00:02.795 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T02:00:02.795 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T02:00:02.795 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T02:00:02.795 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T02:00:02.795 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T02:00:02.796 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T02:00:02.796 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T02:00:02.799 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T02:00:02.799 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T02:00:02.800 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-04-24T02:00:02.908 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:02.902+0000 7f71ead9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:02.911 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:02.911+0000 7f71ead9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:02.928 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T02:00:02.928 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T02:00:02.928 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T02:00:02.928 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T02:00:02.928 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T02:00:02.928 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T02:00:02.929 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T02:00:02.929 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T02:00:02.929 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T02:00:02.929 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T02:00:02.929 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T02:00:02.929 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T02:00:02.931 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T02:00:02.931 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T02:00:02.932 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-04-24T02:00:03.040 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:03.035+0000 7fb41c910640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:03.044 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:03.043+0000 7fb41c910640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:03.059 INFO:tasks.rados.rados.0.smithi079.stdout:1184: done (3 left) 2024-04-24T02:00:03.059 INFO:tasks.rados.rados.0.smithi079.stdout:1185: done (2 left) 2024-04-24T02:00:03.060 INFO:tasks.rados.rados.0.smithi079.stdout:1186: done (1 left) 2024-04-24T02:00:03.060 INFO:tasks.rados.rados.0.smithi079.stdout:1187: done (0 left) 2024-04-24T02:00:03.060 INFO:tasks.rados.rados.0.smithi079.stdout:1188: snap_remove snap 134 2024-04-24T02:00:03.060 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T02:00:03.061 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T02:00:03.061 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T02:00:03.061 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T02:00:03.061 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T02:00:03.061 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T02:00:03.061 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T02:00:03.061 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T02:00:03.061 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T02:00:03.061 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T02:00:03.061 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T02:00:03.061 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T02:00:03.064 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T02:00:03.064 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T02:00:04.061 INFO:tasks.rados.rados.0.smithi079.stdout:1188: done (0 left) 2024-04-24T02:00:04.061 INFO:tasks.rados.rados.0.smithi079.stdout:1189: write (excl) oid 32 current snap is 136 2024-04-24T02:00:04.061 INFO:tasks.rados.rados.0.smithi079.stdout:1189: seq_num 581 ranges {564867=562497,1898744=415196,2755347=725863} 2024-04-24T02:00:04.067 INFO:tasks.rados.rados.0.smithi079.stdout:1189: writing smithi07929762-32 from 564867 to 1127364 tid 1 2024-04-24T02:00:04.074 INFO:tasks.rados.rados.0.smithi079.stdout:1189: writing smithi07929762-32 from 1898744 to 2313940 tid 2 2024-04-24T02:00:04.082 INFO:tasks.rados.rados.0.smithi079.stdout:1189: writing smithi07929762-32 from 2755347 to 3481210 tid 3 2024-04-24T02:00:04.082 INFO:tasks.rados.rados.0.smithi079.stdout:1190: snap_remove snap 132 2024-04-24T02:00:04.579 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T02:00:04.685 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:04.683+0000 7f6348554640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:04.689 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:04.688+0000 7f6348554640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:05.072 INFO:tasks.rados.rados.0.smithi079.stdout:1190: done (1 left) 2024-04-24T02:00:05.072 INFO:tasks.rados.rados.0.smithi079.stdout:1191: snap_remove snap 127 2024-04-24T02:00:06.072 INFO:tasks.rados.rados.0.smithi079.stdout:1191: done (1 left) 2024-04-24T02:00:06.072 INFO:tasks.rados.rados.0.smithi079.stdout:1192: delete oid 37 current snap is 136 2024-04-24T02:00:06.072 INFO:tasks.rados.rados.0.smithi079.stdout:1189: finishing write tid 1 to smithi07929762-32 2024-04-24T02:00:06.072 INFO:tasks.rados.rados.0.smithi079.stdout:1189: finishing write tid 2 to smithi07929762-32 2024-04-24T02:00:06.072 INFO:tasks.rados.rados.0.smithi079.stdout:1189: finishing write tid 3 to smithi07929762-32 2024-04-24T02:00:06.072 INFO:tasks.rados.rados.0.smithi079.stdout:1189: finishing write tid 4 to smithi07929762-32 2024-04-24T02:00:06.072 INFO:tasks.rados.rados.0.smithi079.stdout:1189: finishing write tid 5 to smithi07929762-32 2024-04-24T02:00:06.072 INFO:tasks.rados.rados.0.smithi079.stdout:1189: oid 32 updating version 0 to 212 2024-04-24T02:00:06.072 INFO:tasks.rados.rados.0.smithi079.stdout:1189: oid 32 version 212 is already newer than 212 2024-04-24T02:00:06.073 INFO:tasks.rados.rados.0.smithi079.stdout:1189: oid 32 version 212 is already newer than 212 2024-04-24T02:00:06.073 INFO:tasks.rados.rados.0.smithi079.stdout:1189: oid 32 version 212 is already newer than 209 2024-04-24T02:00:06.073 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 32 v 212 (ObjNum 581 snap 136 seq_num 581) dirty exists 2024-04-24T02:00:06.073 INFO:tasks.rados.rados.0.smithi079.stdout:1189: left oid 32 (ObjNum 581 snap 136 seq_num 581) 2024-04-24T02:00:06.196 INFO:tasks.rados.rados.0.smithi079.stdout:1189: done (1 left) 2024-04-24T02:00:06.196 INFO:tasks.rados.rados.0.smithi079.stdout:1192: done (0 left) 2024-04-24T02:00:06.196 INFO:tasks.rados.rados.0.smithi079.stdout:1193: write oid 25 current snap is 136 2024-04-24T02:00:06.196 INFO:tasks.rados.rados.0.smithi079.stdout:1193: seq_num 583 ranges {598481=712995,1899355=418865,3093840=704641} 2024-04-24T02:00:06.203 INFO:tasks.rados.rados.0.smithi079.stdout:1193: writing smithi07929762-25 from 598481 to 1311476 tid 1 2024-04-24T02:00:06.208 INFO:tasks.rados.rados.0.smithi079.stdout:1193: writing smithi07929762-25 from 1899355 to 2318220 tid 2 2024-04-24T02:00:06.218 INFO:tasks.rados.rados.0.smithi079.stdout:1193: writing smithi07929762-25 from 3093840 to 3798481 tid 3 2024-04-24T02:00:06.218 INFO:tasks.rados.rados.0.smithi079.stdout:1194: read oid 48 snap -1 2024-04-24T02:00:06.218 INFO:tasks.rados.rados.0.smithi079.stdout:1194: expect (ObjNum 553 snap 127 seq_num 553) 2024-04-24T02:00:06.218 INFO:tasks.rados.rados.0.smithi079.stdout:1195: delete oid 7 current snap is 136 2024-04-24T02:00:06.299 INFO:tasks.rados.rados.0.smithi079.stdout:1195: done (2 left) 2024-04-24T02:00:06.299 INFO:tasks.rados.rados.0.smithi079.stdout:1196: write (excl) oid 22 current snap is 136 2024-04-24T02:00:06.299 INFO:tasks.rados.rados.0.smithi079.stdout:1196: seq_num 585 ranges {632095=463493,1616319=422534,2748686=487304} 2024-04-24T02:00:06.306 INFO:tasks.rados.rados.0.smithi079.stdout:1196: writing smithi07929762-22 from 632095 to 1095588 tid 1 2024-04-24T02:00:06.311 INFO:tasks.rados.rados.0.smithi079.stdout:1196: writing smithi07929762-22 from 1616319 to 2038853 tid 2 2024-04-24T02:00:06.319 INFO:tasks.rados.rados.0.smithi079.stdout:1196: writing smithi07929762-22 from 2748686 to 3235990 tid 3 2024-04-24T02:00:06.319 INFO:tasks.rados.rados.0.smithi079.stdout:1197: snap_create 2024-04-24T02:00:06.319 INFO:tasks.rados.rados.0.smithi079.stdout:1193: finishing write tid 1 to smithi07929762-25 2024-04-24T02:00:06.319 INFO:tasks.rados.rados.0.smithi079.stdout:1194: expect (ObjNum 553 snap 127 seq_num 553) 2024-04-24T02:00:06.595 INFO:tasks.rados.rados.0.smithi079.stdout:1193: finishing write tid 2 to smithi07929762-25 2024-04-24T02:00:06.595 INFO:tasks.rados.rados.0.smithi079.stdout:1193: finishing write tid 3 to smithi07929762-25 2024-04-24T02:00:06.596 INFO:tasks.rados.rados.0.smithi079.stdout:1193: finishing write tid 4 to smithi07929762-25 2024-04-24T02:00:06.596 INFO:tasks.rados.rados.0.smithi079.stdout:1193: finishing write tid 5 to smithi07929762-25 2024-04-24T02:00:06.596 INFO:tasks.rados.rados.0.smithi079.stdout:1193: oid 25 updating version 0 to 413 2024-04-24T02:00:06.596 INFO:tasks.rados.rados.0.smithi079.stdout:1193: oid 25 version 413 is already newer than 413 2024-04-24T02:00:06.596 INFO:tasks.rados.rados.0.smithi079.stdout:1193: oid 25 version 413 is already newer than 413 2024-04-24T02:00:06.596 INFO:tasks.rados.rados.0.smithi079.stdout:1193: oid 25 version 413 is already newer than 413 2024-04-24T02:00:06.596 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 25 v 413 (ObjNum 583 snap 136 seq_num 583) dirty exists 2024-04-24T02:00:06.596 INFO:tasks.rados.rados.0.smithi079.stdout:1193: left oid 25 (ObjNum 583 snap 136 seq_num 583) 2024-04-24T02:00:06.596 INFO:tasks.rados.rados.0.smithi079.stdout:1196: finishing write tid 1 to smithi07929762-22 2024-04-24T02:00:06.596 INFO:tasks.rados.rados.0.smithi079.stdout:1196: finishing write tid 2 to smithi07929762-22 2024-04-24T02:00:06.596 INFO:tasks.rados.rados.0.smithi079.stdout:1196: finishing write tid 3 to smithi07929762-22 2024-04-24T02:00:06.596 INFO:tasks.rados.rados.0.smithi079.stdout:1196: finishing write tid 4 to smithi07929762-22 2024-04-24T02:00:06.596 INFO:tasks.rados.rados.0.smithi079.stdout:1196: finishing write tid 5 to smithi07929762-22 2024-04-24T02:00:06.596 INFO:tasks.rados.rados.0.smithi079.stdout:1196: oid 22 updating version 0 to 158 2024-04-24T02:00:06.596 INFO:tasks.rados.rados.0.smithi079.stdout:1196: oid 22 version 158 is already newer than 155 2024-04-24T02:00:06.596 INFO:tasks.rados.rados.0.smithi079.stdout:1196: oid 22 version 158 is already newer than 158 2024-04-24T02:00:06.597 INFO:tasks.rados.rados.0.smithi079.stdout:1196: oid 22 version 158 is already newer than 158 2024-04-24T02:00:06.597 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 22 v 158 (ObjNum 585 snap 136 seq_num 585) dirty exists 2024-04-24T02:00:06.597 INFO:tasks.rados.rados.0.smithi079.stdout:1196: left oid 22 (ObjNum 585 snap 136 seq_num 585) 2024-04-24T02:00:06.783 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T02:00:06.888 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:06.885+0000 7f04c846d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:06.891 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:06.891+0000 7f04c846d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:07.086 INFO:tasks.rados.rados.0.smithi079.stdout:1193: done (3 left) 2024-04-24T02:00:07.086 INFO:tasks.rados.rados.0.smithi079.stdout:1194: done (2 left) 2024-04-24T02:00:07.086 INFO:tasks.rados.rados.0.smithi079.stdout:1196: done (1 left) 2024-04-24T02:00:07.086 INFO:tasks.rados.rados.0.smithi079.stdout:1197: done (0 left) 2024-04-24T02:00:07.086 INFO:tasks.rados.rados.0.smithi079.stdout:1198: read oid 31 snap 131 2024-04-24T02:00:07.086 INFO:tasks.rados.rados.0.smithi079.stdout:1198: expect deleted 2024-04-24T02:00:07.086 INFO:tasks.rados.rados.0.smithi079.stdout:1199: write (excl) oid 45 current snap is 137 2024-04-24T02:00:07.086 INFO:tasks.rados.rados.0.smithi079.stdout:1199: seq_num 586 ranges {648902=655095,2074801=566192,3176109=672793} 2024-04-24T02:00:07.095 INFO:tasks.rados.rados.0.smithi079.stdout:1199: writing smithi07929762-45 from 648902 to 1303997 tid 1 2024-04-24T02:00:07.105 INFO:tasks.rados.rados.0.smithi079.stdout:1199: writing smithi07929762-45 from 2074801 to 2640993 tid 2 2024-04-24T02:00:07.115 INFO:tasks.rados.rados.0.smithi079.stdout:1199: writing smithi07929762-45 from 3176109 to 3848902 tid 3 2024-04-24T02:00:07.115 INFO:tasks.rados.rados.0.smithi079.stdout:1200: write (excl) oid 17 current snap is 137 2024-04-24T02:00:07.115 INFO:tasks.rados.rados.0.smithi079.stdout:1200: seq_num 587 ranges {665709=730344,2133283=426203,3203532=509848} 2024-04-24T02:00:07.123 INFO:tasks.rados.rados.0.smithi079.stdout:1200: writing smithi07929762-17 from 665709 to 1396053 tid 1 2024-04-24T02:00:07.130 INFO:tasks.rados.rados.0.smithi079.stdout:1200: writing smithi07929762-17 from 2133283 to 2559486 tid 2 2024-04-24T02:00:07.136 INFO:tasks.rados.rados.0.smithi079.stdout:1200: writing smithi07929762-17 from 3203532 to 3713380 tid 3 2024-04-24T02:00:07.137 INFO:tasks.rados.rados.0.smithi079.stdout:1201: write oid 2 current snap is 137 2024-04-24T02:00:07.137 INFO:tasks.rados.rados.0.smithi079.stdout:1201: seq_num 588 ranges {682516=405593,1791765=569861,2830955=721120} 2024-04-24T02:00:07.143 INFO:tasks.rados.rados.0.smithi079.stdout:1201: writing smithi07929762-2 from 682516 to 1088109 tid 1 2024-04-24T02:00:07.150 INFO:tasks.rados.rados.0.smithi079.stdout:1201: writing smithi07929762-2 from 1791765 to 2361626 tid 2 2024-04-24T02:00:07.157 INFO:tasks.rados.rados.0.smithi079.stdout:1201: writing smithi07929762-2 from 2830955 to 3552075 tid 3 2024-04-24T02:00:07.157 INFO:tasks.rados.rados.0.smithi079.stdout:1202: write (excl) oid 33 current snap is 137 2024-04-24T02:00:07.157 INFO:tasks.rados.rados.0.smithi079.stdout:1202: seq_num 589 ranges {699323=480842,1850247=429872,2858378=532392} 2024-04-24T02:00:07.163 INFO:tasks.rados.rados.0.smithi079.stdout:1202: writing smithi07929762-33 from 699323 to 1180165 tid 1 2024-04-24T02:00:07.170 INFO:tasks.rados.rados.0.smithi079.stdout:1202: writing smithi07929762-33 from 1850247 to 2280119 tid 2 2024-04-24T02:00:07.176 INFO:tasks.rados.rados.0.smithi079.stdout:1202: writing smithi07929762-33 from 2858378 to 3390770 tid 3 2024-04-24T02:00:07.176 INFO:tasks.rados.rados.0.smithi079.stdout:1203: snap_remove snap 135 2024-04-24T02:00:08.065 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-04-24T02:00:08.066 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-04-24T02:00:08.086 INFO:tasks.rados.rados.0.smithi079.stdout:1203: done (5 left) 2024-04-24T02:00:08.086 INFO:tasks.rados.rados.0.smithi079.stdout:1204: read oid 42 snap -1 2024-04-24T02:00:08.086 INFO:tasks.rados.rados.0.smithi079.stdout:1204: expect deleted 2024-04-24T02:00:08.086 INFO:tasks.rados.rados.0.smithi079.stdout:1205: read oid 22 snap -1 2024-04-24T02:00:08.086 INFO:tasks.rados.rados.0.smithi079.stdout:1205: expect (ObjNum 585 snap 136 seq_num 585) 2024-04-24T02:00:08.086 INFO:tasks.rados.rados.0.smithi079.stdout:1206: delete oid 44 current snap is 137 2024-04-24T02:00:08.086 INFO:tasks.rados.rados.0.smithi079.stdout:1199: finishing write tid 1 to smithi07929762-45 2024-04-24T02:00:08.086 INFO:tasks.rados.rados.0.smithi079.stdout:1200: finishing write tid 1 to smithi07929762-17 2024-04-24T02:00:08.086 INFO:tasks.rados.rados.0.smithi079.stdout:1199: finishing write tid 2 to smithi07929762-45 2024-04-24T02:00:08.087 INFO:tasks.rados.rados.0.smithi079.stdout:1201: finishing write tid 1 to smithi07929762-2 2024-04-24T02:00:08.087 INFO:tasks.rados.rados.0.smithi079.stdout:1199: finishing write tid 3 to smithi07929762-45 2024-04-24T02:00:08.087 INFO:tasks.rados.rados.0.smithi079.stdout:1199: finishing write tid 4 to smithi07929762-45 2024-04-24T02:00:08.087 INFO:tasks.rados.rados.0.smithi079.stdout:1199: finishing write tid 5 to smithi07929762-45 2024-04-24T02:00:08.087 INFO:tasks.rados.rados.0.smithi079.stdout:1199: oid 45 updating version 0 to 322 2024-04-24T02:00:08.087 INFO:tasks.rados.rados.0.smithi079.stdout:1199: oid 45 version 322 is already newer than 322 2024-04-24T02:00:08.087 INFO:tasks.rados.rados.0.smithi079.stdout:1199: oid 45 version 322 is already newer than 319 2024-04-24T02:00:08.087 INFO:tasks.rados.rados.0.smithi079.stdout:1199: oid 45 version 322 is already newer than 322 2024-04-24T02:00:08.087 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 45 v 322 (ObjNum 586 snap 137 seq_num 586) dirty exists 2024-04-24T02:00:08.087 INFO:tasks.rados.rados.0.smithi079.stdout:1199: left oid 45 (ObjNum 586 snap 137 seq_num 586) 2024-04-24T02:00:08.087 INFO:tasks.rados.rados.0.smithi079.stdout:1201: finishing write tid 2 to smithi07929762-2 2024-04-24T02:00:08.087 INFO:tasks.rados.rados.0.smithi079.stdout:1202: finishing write tid 1 to smithi07929762-33 2024-04-24T02:00:08.087 INFO:tasks.rados.rados.0.smithi079.stdout:1201: finishing write tid 3 to smithi07929762-2 2024-04-24T02:00:08.087 INFO:tasks.rados.rados.0.smithi079.stdout:1200: finishing write tid 2 to smithi07929762-17 2024-04-24T02:00:08.087 INFO:tasks.rados.rados.0.smithi079.stdout:1201: finishing write tid 4 to smithi07929762-2 2024-04-24T02:00:08.088 INFO:tasks.rados.rados.0.smithi079.stdout:1201: finishing write tid 5 to smithi07929762-2 2024-04-24T02:00:08.088 INFO:tasks.rados.rados.0.smithi079.stdout:1201: oid 2 updating version 0 to 77 2024-04-24T02:00:08.088 INFO:tasks.rados.rados.0.smithi079.stdout:1201: oid 2 version 77 is already newer than 77 2024-04-24T02:00:08.088 INFO:tasks.rados.rados.0.smithi079.stdout:1201: oid 2 version 77 is already newer than 77 2024-04-24T02:00:08.088 INFO:tasks.rados.rados.0.smithi079.stdout:1201: oid 2 version 77 is already newer than 77 2024-04-24T02:00:08.088 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 2 v 77 (ObjNum 588 snap 137 seq_num 588) dirty exists 2024-04-24T02:00:08.088 INFO:tasks.rados.rados.0.smithi079.stdout:1201: left oid 2 (ObjNum 588 snap 137 seq_num 588) 2024-04-24T02:00:08.088 INFO:tasks.rados.rados.0.smithi079.stdout:1200: finishing write tid 3 to smithi07929762-17 2024-04-24T02:00:08.088 INFO:tasks.rados.rados.0.smithi079.stdout:1202: finishing write tid 2 to smithi07929762-33 2024-04-24T02:00:08.088 INFO:tasks.rados.rados.0.smithi079.stdout:1200: finishing write tid 4 to smithi07929762-17 2024-04-24T02:00:08.088 INFO:tasks.rados.rados.0.smithi079.stdout:1200: finishing write tid 5 to smithi07929762-17 2024-04-24T02:00:08.088 INFO:tasks.rados.rados.0.smithi079.stdout:1200: oid 17 updating version 0 to 323 2024-04-24T02:00:08.088 INFO:tasks.rados.rados.0.smithi079.stdout:1200: oid 17 version 323 is already newer than 320 2024-04-24T02:00:08.088 INFO:tasks.rados.rados.0.smithi079.stdout:1200: oid 17 version 323 is already newer than 323 2024-04-24T02:00:08.089 INFO:tasks.rados.rados.0.smithi079.stdout:1200: oid 17 version 323 is already newer than 323 2024-04-24T02:00:08.089 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 17 v 323 (ObjNum 587 snap 137 seq_num 587) dirty exists 2024-04-24T02:00:08.089 INFO:tasks.rados.rados.0.smithi079.stdout:1200: left oid 17 (ObjNum 587 snap 137 seq_num 587) 2024-04-24T02:00:08.089 INFO:tasks.rados.rados.0.smithi079.stdout:1202: finishing write tid 3 to smithi07929762-33 2024-04-24T02:00:08.089 INFO:tasks.rados.rados.0.smithi079.stdout:1202: finishing write tid 4 to smithi07929762-33 2024-04-24T02:00:08.089 INFO:tasks.rados.rados.0.smithi079.stdout:1202: finishing write tid 5 to smithi07929762-33 2024-04-24T02:00:08.089 INFO:tasks.rados.rados.0.smithi079.stdout:1202: oid 33 updating version 0 to 374 2024-04-24T02:00:08.089 INFO:tasks.rados.rados.0.smithi079.stdout:1202: oid 33 updating version 374 to 377 2024-04-24T02:00:08.089 INFO:tasks.rados.rados.0.smithi079.stdout:1202: oid 33 version 377 is already newer than 377 2024-04-24T02:00:08.089 INFO:tasks.rados.rados.0.smithi079.stdout:1202: oid 33 version 377 is already newer than 377 2024-04-24T02:00:08.089 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 33 v 377 (ObjNum 589 snap 137 seq_num 589) dirty exists 2024-04-24T02:00:08.089 INFO:tasks.rados.rados.0.smithi079.stdout:1202: left oid 33 (ObjNum 589 snap 137 seq_num 589) 2024-04-24T02:00:08.148 INFO:tasks.rados.rados.0.smithi079.stdout:1198: done (7 left) 2024-04-24T02:00:08.149 INFO:tasks.rados.rados.0.smithi079.stdout:1199: done (6 left) 2024-04-24T02:00:08.149 INFO:tasks.rados.rados.0.smithi079.stdout:1200: done (5 left) 2024-04-24T02:00:08.149 INFO:tasks.rados.rados.0.smithi079.stdout:1201: done (4 left) 2024-04-24T02:00:08.149 INFO:tasks.rados.rados.0.smithi079.stdout:1202: done (3 left) 2024-04-24T02:00:08.149 INFO:tasks.rados.rados.0.smithi079.stdout:1204: done (2 left) 2024-04-24T02:00:08.149 INFO:tasks.rados.rados.0.smithi079.stdout:1206: done (1 left) 2024-04-24T02:00:08.149 INFO:tasks.rados.rados.0.smithi079.stdout:1207: read oid 6 snap -1 2024-04-24T02:00:08.149 INFO:tasks.rados.rados.0.smithi079.stdout:1207: expect (ObjNum 567 snap 130 seq_num 567) 2024-04-24T02:00:08.149 INFO:tasks.rados.rados.0.smithi079.stdout:1208: snap_create 2024-04-24T02:00:08.164 INFO:tasks.rados.rados.0.smithi079.stdout:1205: expect (ObjNum 585 snap 136 seq_num 585) 2024-04-24T02:00:08.194 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:08.186+0000 7f57a68ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:08.196 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:08.196+0000 7f57a68ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:08.327 INFO:tasks.rados.rados.0.smithi079.stdout:1207: expect (ObjNum 567 snap 130 seq_num 567) 2024-04-24T02:00:08.984 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T02:00:09.086 INFO:teuthology.orchestra.run.smithi079.stderr:reweighted osd.0 to 1 (10000) 2024-04-24T02:00:09.090 INFO:tasks.rados.rados.0.smithi079.stdout:1205: done (2 left) 2024-04-24T02:00:09.090 INFO:tasks.rados.rados.0.smithi079.stdout:1207: done (1 left) 2024-04-24T02:00:09.090 INFO:tasks.rados.rados.0.smithi079.stdout:1208: done (0 left) 2024-04-24T02:00:09.090 INFO:tasks.rados.rados.0.smithi079.stdout:1209: write (excl) oid 7 current snap is 138 2024-04-24T02:00:09.090 INFO:tasks.rados.rados.0.smithi079.stdout:1209: seq_num 591 ranges {732937=631340,1850858=433541,2796871=671289,3888195=44742} 2024-04-24T02:00:09.092 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:09.090+0000 7f22f6c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:09.097 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:09.096+0000 7f22f6c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:09.097 INFO:tasks.rados.rados.0.smithi079.stdout:1209: writing smithi07929762-7 from 732937 to 1364277 tid 1 2024-04-24T02:00:09.100 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-04-24T02:00:09.103 INFO:tasks.rados.rados.0.smithi079.stdout:1209: writing smithi07929762-7 from 1850858 to 2284399 tid 2 2024-04-24T02:00:09.110 INFO:tasks.rados.rados.0.smithi079.stdout:1209: writing smithi07929762-7 from 2796871 to 3468160 tid 3 2024-04-24T02:00:09.112 INFO:tasks.rados.rados.0.smithi079.stdout:1209: writing smithi07929762-7 from 3888195 to 3932937 tid 4 2024-04-24T02:00:09.112 INFO:tasks.rados.rados.0.smithi079.stdout:1210: delete oid 36 current snap is 138 2024-04-24T02:00:09.157 INFO:tasks.rados.rados.0.smithi079.stdout:1209: finishing write tid 1 to smithi07929762-7 2024-04-24T02:00:09.194 INFO:tasks.rados.rados.0.smithi079.stdout:1209: finishing write tid 2 to smithi07929762-7 2024-04-24T02:00:09.206 INFO:tasks.rados.rados.0.smithi079.stdout:1209: finishing write tid 3 to smithi07929762-7 2024-04-24T02:00:09.211 INFO:tasks.rados.rados.0.smithi079.stdout:1209: finishing write tid 4 to smithi07929762-7 2024-04-24T02:00:09.215 INFO:tasks.rados.rados.0.smithi079.stdout:1209: finishing write tid 5 to smithi07929762-7 2024-04-24T02:00:09.215 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:09.209+0000 7f2a03110640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:09.216 INFO:tasks.rados.rados.0.smithi079.stdout:1209: finishing write tid 6 to smithi07929762-7 2024-04-24T02:00:09.216 INFO:tasks.rados.rados.0.smithi079.stdout:1209: oid 7 updating version 0 to 270 2024-04-24T02:00:09.216 INFO:tasks.rados.rados.0.smithi079.stdout:1209: oid 7 version 270 is already newer than 266 2024-04-24T02:00:09.216 INFO:tasks.rados.rados.0.smithi079.stdout:1209: oid 7 version 270 is already newer than 270 2024-04-24T02:00:09.217 INFO:tasks.rados.rados.0.smithi079.stdout:1209: oid 7 version 270 is already newer than 270 2024-04-24T02:00:09.217 INFO:tasks.rados.rados.0.smithi079.stdout:1209: oid 7 version 270 is already newer than 270 2024-04-24T02:00:09.217 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 7 v 270 (ObjNum 591 snap 138 seq_num 591) dirty exists 2024-04-24T02:00:09.217 INFO:tasks.rados.rados.0.smithi079.stdout:1209: left oid 7 (ObjNum 591 snap 138 seq_num 591) 2024-04-24T02:00:09.219 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:09.218+0000 7f2a03110640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:09.221 INFO:tasks.rados.rados.0.smithi079.stdout:1209: done (1 left) 2024-04-24T02:00:09.221 INFO:tasks.rados.rados.0.smithi079.stdout:1210: done (0 left) 2024-04-24T02:00:09.221 INFO:tasks.rados.rados.0.smithi079.stdout:1211: write (excl) oid 13 current snap is 138 2024-04-24T02:00:09.221 INFO:tasks.rados.rados.0.smithi079.stdout:1211: seq_num 593 ranges {766551=498191,1684175=720857,2851717=693833} 2024-04-24T02:00:09.228 INFO:tasks.rados.rados.0.smithi079.stdout:1211: writing smithi07929762-13 from 766551 to 1264742 tid 1 2024-04-24T02:00:09.237 INFO:tasks.rados.rados.0.smithi079.stdout:1211: writing smithi07929762-13 from 1684175 to 2405032 tid 2 2024-04-24T02:00:09.247 INFO:tasks.rados.rados.0.smithi079.stdout:1211: writing smithi07929762-13 from 2851717 to 3545550 tid 3 2024-04-24T02:00:09.247 INFO:tasks.rados.rados.0.smithi079.stdout:1212: delete oid 16 current snap is 138 2024-04-24T02:00:09.333 INFO:tasks.rados.rados.0.smithi079.stdout:1211: finishing write tid 1 to smithi07929762-13 2024-04-24T02:00:09.372 INFO:tasks.rados.rados.0.smithi079.stdout:1211: finishing write tid 2 to smithi07929762-13 2024-04-24T02:00:09.388 INFO:tasks.rados.rados.0.smithi079.stdout:1211: finishing write tid 3 to smithi07929762-13 2024-04-24T02:00:09.389 INFO:tasks.rados.rados.0.smithi079.stdout:1212: done (1 left) 2024-04-24T02:00:09.389 INFO:tasks.rados.rados.0.smithi079.stdout:1213: write oid 41 current snap is 138 2024-04-24T02:00:09.389 INFO:tasks.rados.rados.0.smithi079.stdout:1213: seq_num 595 ranges {400165=648689,1684786=315379} 2024-04-24T02:00:09.395 INFO:tasks.rados.rados.0.smithi079.stdout:1213: writing smithi07929762-41 from 400165 to 1048854 tid 1 2024-04-24T02:00:09.401 INFO:tasks.rados.rados.0.smithi079.stdout:1213: writing smithi07929762-41 from 1684786 to 2000165 tid 2 2024-04-24T02:00:09.401 INFO:tasks.rados.rados.0.smithi079.stdout:1214: read oid 43 snap -1 2024-04-24T02:00:09.401 INFO:tasks.rados.rados.0.smithi079.stdout:1214: expect (ObjNum 548 snap 123 seq_num 548) 2024-04-24T02:00:09.402 INFO:tasks.rados.rados.0.smithi079.stdout:1215: read oid 21 snap -1 2024-04-24T02:00:09.402 INFO:tasks.rados.rados.0.smithi079.stdout:1215: expect (ObjNum 500 snap 112 seq_num 500) 2024-04-24T02:00:09.402 INFO:tasks.rados.rados.0.smithi079.stdout:1216: write oid 10 current snap is 138 2024-04-24T02:00:09.402 INFO:tasks.rados.rados.0.smithi079.stdout:1216: seq_num 596 ranges {416972=723938,1743268=273704} 2024-04-24T02:00:09.411 INFO:tasks.rados.rados.0.smithi079.stdout:1216: writing smithi07929762-10 from 416972 to 1140910 tid 1 2024-04-24T02:00:09.418 INFO:tasks.rados.rados.0.smithi079.stdout:1216: writing smithi07929762-10 from 1743268 to 2016972 tid 2 2024-04-24T02:00:09.418 INFO:tasks.rados.rados.0.smithi079.stdout:1217: read oid 20 snap 136 2024-04-24T02:00:09.418 INFO:tasks.rados.rados.0.smithi079.stdout:1217: expect (ObjNum 565 snap 130 seq_num 565) 2024-04-24T02:00:09.419 INFO:tasks.rados.rados.0.smithi079.stdout:1218: snap_create 2024-04-24T02:00:09.419 INFO:tasks.rados.rados.0.smithi079.stdout:1211: finishing write tid 4 to smithi07929762-13 2024-04-24T02:00:09.419 INFO:tasks.rados.rados.0.smithi079.stdout:1211: finishing write tid 5 to smithi07929762-13 2024-04-24T02:00:09.419 INFO:tasks.rados.rados.0.smithi079.stdout:1211: oid 13 updating version 0 to 422 2024-04-24T02:00:09.419 INFO:tasks.rados.rados.0.smithi079.stdout:1211: oid 13 version 422 is already newer than 422 2024-04-24T02:00:09.419 INFO:tasks.rados.rados.0.smithi079.stdout:1211: oid 13 version 422 is already newer than 419 2024-04-24T02:00:09.419 INFO:tasks.rados.rados.0.smithi079.stdout:1211: oid 13 version 422 is already newer than 422 2024-04-24T02:00:09.419 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 13 v 422 (ObjNum 593 snap 138 seq_num 593) dirty exists 2024-04-24T02:00:09.419 INFO:tasks.rados.rados.0.smithi079.stdout:1211: left oid 13 (ObjNum 593 snap 138 seq_num 593) 2024-04-24T02:00:09.450 INFO:tasks.rados.rados.0.smithi079.stdout:1215: expect (ObjNum 500 snap 112 seq_num 500) 2024-04-24T02:00:09.549 INFO:tasks.rados.rados.0.smithi079.stdout:1217: expect (ObjNum 565 snap 130 seq_num 565) 2024-04-24T02:00:09.762 INFO:tasks.rados.rados.0.smithi079.stdout:1213: finishing write tid 1 to smithi07929762-41 2024-04-24T02:00:09.762 INFO:tasks.rados.rados.0.smithi079.stdout:1216: finishing write tid 1 to smithi07929762-10 2024-04-24T02:00:09.762 INFO:tasks.rados.rados.0.smithi079.stdout:1213: finishing write tid 2 to smithi07929762-41 2024-04-24T02:00:09.762 INFO:tasks.rados.rados.0.smithi079.stdout:1214: expect (ObjNum 548 snap 123 seq_num 548) 2024-04-24T02:00:09.925 INFO:tasks.rados.rados.0.smithi079.stdout:1213: finishing write tid 3 to smithi07929762-41 2024-04-24T02:00:09.925 INFO:tasks.rados.rados.0.smithi079.stdout:1213: finishing write tid 4 to smithi07929762-41 2024-04-24T02:00:09.925 INFO:tasks.rados.rados.0.smithi079.stdout:1213: oid 41 updating version 0 to 162 2024-04-24T02:00:09.925 INFO:tasks.rados.rados.0.smithi079.stdout:1213: oid 41 version 162 is already newer than 162 2024-04-24T02:00:09.925 INFO:tasks.rados.rados.0.smithi079.stdout:1213: oid 41 version 162 is already newer than 162 2024-04-24T02:00:09.925 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 41 v 162 (ObjNum 595 snap 138 seq_num 595) dirty exists 2024-04-24T02:00:09.925 INFO:tasks.rados.rados.0.smithi079.stdout:1213: left oid 41 (ObjNum 595 snap 138 seq_num 595) 2024-04-24T02:00:09.925 INFO:tasks.rados.rados.0.smithi079.stdout:1216: finishing write tid 2 to smithi07929762-10 2024-04-24T02:00:09.926 INFO:tasks.rados.rados.0.smithi079.stdout:1216: finishing write tid 3 to smithi07929762-10 2024-04-24T02:00:09.926 INFO:tasks.rados.rados.0.smithi079.stdout:1216: finishing write tid 4 to smithi07929762-10 2024-04-24T02:00:09.926 INFO:tasks.rados.rados.0.smithi079.stdout:1216: oid 10 updating version 0 to 327 2024-04-24T02:00:09.926 INFO:tasks.rados.rados.0.smithi079.stdout:1216: oid 10 version 327 is already newer than 327 2024-04-24T02:00:09.926 INFO:tasks.rados.rados.0.smithi079.stdout:1216: oid 10 version 327 is already newer than 327 2024-04-24T02:00:09.926 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 10 v 327 (ObjNum 596 snap 138 seq_num 596) dirty exists 2024-04-24T02:00:09.926 INFO:tasks.rados.rados.0.smithi079.stdout:1216: left oid 10 (ObjNum 596 snap 138 seq_num 596) 2024-04-24T02:00:10.089 INFO:teuthology.orchestra.run.smithi079.stderr:reweighted osd.1 to 1 (10000) 2024-04-24T02:00:10.091 INFO:tasks.rados.rados.0.smithi079.stdout:1211: done (6 left) 2024-04-24T02:00:10.092 INFO:tasks.rados.rados.0.smithi079.stdout:1213: done (5 left) 2024-04-24T02:00:10.092 INFO:tasks.rados.rados.0.smithi079.stdout:1214: done (4 left) 2024-04-24T02:00:10.092 INFO:tasks.rados.rados.0.smithi079.stdout:1215: done (3 left) 2024-04-24T02:00:10.092 INFO:tasks.rados.rados.0.smithi079.stdout:1216: done (2 left) 2024-04-24T02:00:10.092 INFO:tasks.rados.rados.0.smithi079.stdout:1217: done (1 left) 2024-04-24T02:00:10.092 INFO:tasks.rados.rados.0.smithi079.stdout:1218: done (0 left) 2024-04-24T02:00:10.092 INFO:tasks.rados.rados.0.smithi079.stdout:1219: delete oid 41 current snap is 139 2024-04-24T02:00:10.105 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-04-24T02:00:10.147 INFO:tasks.rados.rados.0.smithi079.stdout:1219: done (0 left) 2024-04-24T02:00:10.147 INFO:tasks.rados.rados.0.smithi079.stdout:1220: read oid 43 snap -1 2024-04-24T02:00:10.147 INFO:tasks.rados.rados.0.smithi079.stdout:1220: expect (ObjNum 548 snap 123 seq_num 548) 2024-04-24T02:00:10.147 INFO:tasks.rados.rados.0.smithi079.stdout:1221: delete oid 18 current snap is 139 2024-04-24T02:00:10.174 INFO:tasks.rados.rados.0.smithi079.stdout:1221: done (1 left) 2024-04-24T02:00:10.174 INFO:tasks.rados.rados.0.smithi079.stdout:1222: read oid 17 snap 136 2024-04-24T02:00:10.175 INFO:tasks.rados.rados.0.smithi079.stdout:1222: expect deleted 2024-04-24T02:00:10.175 INFO:tasks.rados.rados.0.smithi079.stdout:1223: write oid 10 current snap is 139 2024-04-24T02:00:10.175 INFO:tasks.rados.rados.0.smithi079.stdout:1223: seq_num 599 ranges {467393=549685,1802361=265032} 2024-04-24T02:00:10.180 INFO:tasks.rados.rados.0.smithi079.stdout:1223: writing smithi07929762-10 from 467393 to 1017078 tid 1 2024-04-24T02:00:10.186 INFO:tasks.rados.rados.0.smithi079.stdout:1223: writing smithi07929762-10 from 1802361 to 2067393 tid 2 2024-04-24T02:00:10.186 INFO:tasks.rados.rados.0.smithi079.stdout:1224: snap_create 2024-04-24T02:00:10.186 INFO:tasks.rados.rados.0.smithi079.stdout:1220: expect (ObjNum 548 snap 123 seq_num 548) 2024-04-24T02:00:10.266 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:10.259+0000 7ff53619d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:10.270 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:10.269+0000 7ff53619d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:10.377 INFO:tasks.rados.rados.0.smithi079.stdout:1223: finishing write tid 1 to smithi07929762-10 2024-04-24T02:00:10.377 INFO:tasks.rados.rados.0.smithi079.stdout:1223: finishing write tid 2 to smithi07929762-10 2024-04-24T02:00:10.378 INFO:tasks.rados.rados.0.smithi079.stdout:1223: finishing write tid 3 to smithi07929762-10 2024-04-24T02:00:10.378 INFO:tasks.rados.rados.0.smithi079.stdout:1223: finishing write tid 4 to smithi07929762-10 2024-04-24T02:00:10.378 INFO:tasks.rados.rados.0.smithi079.stdout:1223: oid 10 updating version 0 to 331 2024-04-24T02:00:10.378 INFO:tasks.rados.rados.0.smithi079.stdout:1223: oid 10 version 331 is already newer than 331 2024-04-24T02:00:10.378 INFO:tasks.rados.rados.0.smithi079.stdout:1223: oid 10 version 331 is already newer than 331 2024-04-24T02:00:10.378 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 10 v 331 (ObjNum 599 snap 139 seq_num 599) dirty exists 2024-04-24T02:00:10.378 INFO:tasks.rados.rados.0.smithi079.stdout:1223: left oid 10 (ObjNum 599 snap 139 seq_num 599) 2024-04-24T02:00:11.054 INFO:teuthology.orchestra.run.smithi079.stderr:reweighted osd.2 to 1 (10000) 2024-04-24T02:00:11.057 INFO:tasks.rados.rados.0.smithi079.stdout:1220: done (3 left) 2024-04-24T02:00:11.057 INFO:tasks.rados.rados.0.smithi079.stdout:1222: done (2 left) 2024-04-24T02:00:11.057 INFO:tasks.rados.rados.0.smithi079.stdout:1223: done (1 left) 2024-04-24T02:00:11.057 INFO:tasks.rados.rados.0.smithi079.stdout:1224: done (0 left) 2024-04-24T02:00:11.057 INFO:tasks.rados.rados.0.smithi079.stdout:1225: read oid 28 snap 137 2024-04-24T02:00:11.057 INFO:tasks.rados.rados.0.smithi079.stdout:1225: expect (ObjNum 575 snap 131 seq_num 575) 2024-04-24T02:00:11.057 INFO:tasks.rados.rados.0.smithi079.stdout:1226: snap_remove snap 139 2024-04-24T02:00:11.070 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-04-24T02:00:11.220 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:11.213+0000 7f29e9a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:11.223 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:11.223+0000 7f29e9a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:11.234 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T02:00:11.339 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:11.337+0000 7fc80fced640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:11.343 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:11.342+0000 7fc80fced640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:12.055 INFO:teuthology.orchestra.run.smithi079.stderr:reweighted osd.3 to 1 (10000) 2024-04-24T02:00:12.061 INFO:tasks.rados.rados.0.smithi079.stdout:1226: done (1 left) 2024-04-24T02:00:12.061 INFO:tasks.rados.rados.0.smithi079.stdout:1227: read oid 46 snap -1 2024-04-24T02:00:12.061 INFO:tasks.rados.rados.0.smithi079.stdout:1227: expect deleted 2024-04-24T02:00:12.061 INFO:tasks.rados.rados.0.smithi079.stdout:1228: read oid 14 snap -1 2024-04-24T02:00:12.061 INFO:tasks.rados.rados.0.smithi079.stdout:1228: expect (ObjNum 506 snap 112 seq_num 506) 2024-04-24T02:00:12.061 INFO:tasks.rados.rados.0.smithi079.stdout:1229: read oid 29 snap 138 2024-04-24T02:00:12.061 INFO:tasks.rados.rados.0.smithi079.stdout:1229: expect deleted 2024-04-24T02:00:12.061 INFO:tasks.rados.rados.0.smithi079.stdout:1230: delete oid 1 current snap is 140 2024-04-24T02:00:12.061 INFO:tasks.rados.rados.0.smithi079.stdout:1225: expect (ObjNum 575 snap 131 seq_num 575) 2024-04-24T02:00:12.074 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-04-24T02:00:12.075 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T02:00:12.227 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:12.222+0000 7fcf5d26d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:12.231 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:12.230+0000 7fcf5d26d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:12.240 INFO:tasks.rados.rados.0.smithi079.stdout:1225: done (4 left) 2024-04-24T02:00:12.240 INFO:tasks.rados.rados.0.smithi079.stdout:1230: done (3 left) 2024-04-24T02:00:12.240 INFO:tasks.rados.rados.0.smithi079.stdout:1231: write oid 13 current snap is 140 2024-04-24T02:00:12.240 INFO:tasks.rados.rados.0.smithi079.stdout:1228: expect (ObjNum 506 snap 112 seq_num 506) 2024-04-24T02:00:12.338 INFO:tasks.rados.rados.0.smithi079.stdout:1231: seq_num 601 ranges {501007=416536,1635678=465329} 2024-04-24T02:00:12.343 INFO:tasks.rados.rados.0.smithi079.stdout:1231: writing smithi07929762-13 from 501007 to 917543 tid 1 2024-04-24T02:00:12.351 INFO:tasks.rados.rados.0.smithi079.stdout:1231: writing smithi07929762-13 from 1635678 to 2101007 tid 2 2024-04-24T02:00:12.351 INFO:tasks.rados.rados.0.smithi079.stdout:1227: done (3 left) 2024-04-24T02:00:12.351 INFO:tasks.rados.rados.0.smithi079.stdout:1228: done (2 left) 2024-04-24T02:00:12.351 INFO:tasks.rados.rados.0.smithi079.stdout:1229: done (1 left) 2024-04-24T02:00:12.351 INFO:tasks.rados.rados.0.smithi079.stdout:1232: write oid 22 current snap is 140 2024-04-24T02:00:12.351 INFO:tasks.rados.rados.0.smithi079.stdout:1232: seq_num 602 ranges {517814=491785,1577807=540007} 2024-04-24T02:00:12.357 INFO:tasks.rados.rados.0.smithi079.stdout:1232: writing smithi07929762-22 from 517814 to 1009599 tid 1 2024-04-24T02:00:12.367 INFO:tasks.rados.rados.0.smithi079.stdout:1232: writing smithi07929762-22 from 1577807 to 2117814 tid 2 2024-04-24T02:00:12.367 INFO:tasks.rados.rados.0.smithi079.stdout:1233: snap_create 2024-04-24T02:00:12.384 INFO:tasks.rados.rados.0.smithi079.stdout:1231: finishing write tid 1 to smithi07929762-13 2024-04-24T02:00:12.401 INFO:tasks.rados.rados.0.smithi079.stdout:1231: finishing write tid 2 to smithi07929762-13 2024-04-24T02:00:12.425 INFO:tasks.rados.rados.0.smithi079.stdout:1231: finishing write tid 3 to smithi07929762-13 2024-04-24T02:00:12.426 INFO:tasks.rados.rados.0.smithi079.stdout:1231: finishing write tid 4 to smithi07929762-13 2024-04-24T02:00:12.426 INFO:tasks.rados.rados.0.smithi079.stdout:1231: oid 13 updating version 0 to 426 2024-04-24T02:00:12.426 INFO:tasks.rados.rados.0.smithi079.stdout:1231: oid 13 version 426 is already newer than 426 2024-04-24T02:00:12.426 INFO:tasks.rados.rados.0.smithi079.stdout:1231: oid 13 version 426 is already newer than 426 2024-04-24T02:00:12.426 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 13 v 426 (ObjNum 601 snap 140 seq_num 601) dirty exists 2024-04-24T02:00:12.426 INFO:tasks.rados.rados.0.smithi079.stdout:1231: left oid 13 (ObjNum 601 snap 140 seq_num 601) 2024-04-24T02:00:12.428 INFO:tasks.rados.rados.0.smithi079.stdout:1232: finishing write tid 1 to smithi07929762-22 2024-04-24T02:00:12.429 INFO:tasks.rados.rados.0.smithi079.stdout:1232: finishing write tid 2 to smithi07929762-22 2024-04-24T02:00:12.432 INFO:tasks.rados.rados.0.smithi079.stdout:1232: finishing write tid 3 to smithi07929762-22 2024-04-24T02:00:12.433 INFO:tasks.rados.rados.0.smithi079.stdout:1232: finishing write tid 4 to smithi07929762-22 2024-04-24T02:00:12.433 INFO:tasks.rados.rados.0.smithi079.stdout:1232: oid 22 updating version 0 to 168 2024-04-24T02:00:12.433 INFO:tasks.rados.rados.0.smithi079.stdout:1232: oid 22 version 168 is already newer than 168 2024-04-24T02:00:12.433 INFO:tasks.rados.rados.0.smithi079.stdout:1232: oid 22 version 168 is already newer than 168 2024-04-24T02:00:12.433 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 22 v 168 (ObjNum 602 snap 140 seq_num 602) dirty exists 2024-04-24T02:00:12.433 INFO:tasks.rados.rados.0.smithi079.stdout:1232: left oid 22 (ObjNum 602 snap 140 seq_num 602) 2024-04-24T02:00:12.451 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T02:00:12.451 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-24T02:00:12.464 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":435,"stamp":"2024-04-24T02:00:12.058449+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":119368836,"num_objects":51,"num_object_clones":11,"num_object_copies":153,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":2246,"num_read_kb":1531033,"num_write":2123,"num_write_kb":545936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4073,"ondisk_log_size":4073,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":390076,"kb_used_data":248060,"kb_used_omap":0,"kb_used_meta":140544,"kb_avail":377097284,"statfs":{"total":386547056640,"available":386147618816,"internally_reserved":0,"allocated":254013440,"data_stored":267232518,"data_compressed":89889,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":143917056},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5815222,"num_objects":-2,"num_object_clones":-3,"num_object_copies":-6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":31,"num_read_kb":25382,"num_write":37,"num_write_kb":11428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.975360"},"pg_stats":[{"pgid":"3.f","version":"289'214","reported_seq":1096,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-04-24T02:00:07.116472+0000","last_change":"2024-04-24T02:00:07.116472+0000","last_active":"2024-04-24T02:00:07.116472+0000","last_peered":"2024-04-24T02:00:07.116472+0000","last_clean":"2024-04-24T02:00:07.116472+0000","last_became_active":"2024-04-24T01:55:30.386726+0000","last_became_peered":"2024-04-24T01:55:30.386726+0000","last_unstale":"2024-04-24T02:00:07.116472+0000","last_undegraded":"2024-04-24T02:00:07.116472+0000","last_fullsized":"2024-04-24T02:00:07.116472+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":214,"log_dups_size":0,"ondisk_log_size":214,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":20,"snaptrim_duration":0,"stat_sum":{"num_bytes":7059138,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":163,"num_read_kb":147337,"num_write":103,"num_write_kb":30109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"291'324","reported_seq":1154,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-04-24T02:00:09.100093+0000","last_change":"2024-04-24T02:00:08.152477+0000","last_active":"2024-04-24T02:00:09.100093+0000","last_peered":"2024-04-24T02:00:09.100093+0000","last_clean":"2024-04-24T02:00:09.100093+0000","last_became_active":"2024-04-24T01:55:30.321866+0000","last_became_peered":"2024-04-24T01:55:30.321866+0000","last_unstale":"2024-04-24T02:00:09.100093+0000","last_undegraded":"2024-04-24T02:00:09.100093+0000","last_fullsized":"2024-04-24T02:00:09.100093+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":324,"log_dups_size":0,"ondisk_log_size":324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":30,"snaptrim_duration":0,"stat_sum":{"num_bytes":9916427,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":175,"num_read_kb":116839,"num_write":161,"num_write_kb":44263,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"264'229","reported_seq":1071,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-24T02:00:08.096053+0000","last_change":"2024-04-24T02:00:08.102657+0000","last_active":"2024-04-24T02:00:08.096053+0000","last_peered":"2024-04-24T02:00:08.096053+0000","last_clean":"2024-04-24T02:00:08.096053+0000","last_became_active":"2024-04-24T01:55:30.320186+0000","last_became_peered":"2024-04-24T01:55:30.320186+0000","last_unstale":"2024-04-24T02:00:08.096053+0000","last_undegraded":"2024-04-24T02:00:08.096053+0000","last_fullsized":"2024-04-24T02:00:08.096053+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":229,"log_dups_size":0,"ondisk_log_size":229,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":24,"snaptrim_duration":0,"stat_sum":{"num_bytes":5159914,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":125,"num_read_kb":85296,"num_write":124,"num_write_kb":34018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"289'282","reported_seq":1137,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-04-24T02:00:07.117089+0000","last_change":"2024-04-24T02:00:07.117089+0000","last_active":"2024-04-24T02:00:07.117089+0000","last_peered":"2024-04-24T02:00:07.117089+0000","last_clean":"2024-04-24T02:00:07.117089+0000","last_became_active":"2024-04-24T01:55:30.288081+0000","last_became_peered":"2024-04-24T01:55:30.288081+0000","last_unstale":"2024-04-24T02:00:07.117089+0000","last_undegraded":"2024-04-24T02:00:07.117089+0000","last_fullsized":"2024-04-24T02:00:07.117089+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":282,"log_dups_size":0,"ondisk_log_size":282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":28,"snaptrim_duration":0,"stat_sum":{"num_bytes":6807033,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":150,"num_read_kb":109442,"num_write":139,"num_write_kb":37310,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"277'107","reported_seq":969,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-04-24T02:00:09.144222+0000","last_change":"2024-04-24T02:00:08.124445+0000","last_active":"2024-04-24T02:00:09.144222+0000","last_peered":"2024-04-24T02:00:09.144222+0000","last_clean":"2024-04-24T02:00:09.144222+0000","last_became_active":"2024-04-24T01:55:30.333815+0000","last_became_peered":"2024-04-24T01:55:30.333815+0000","last_unstale":"2024-04-24T02:00:09.144222+0000","last_undegraded":"2024-04-24T02:00:09.144222+0000","last_fullsized":"2024-04-24T02:00:09.144222+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":12,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":59,"num_read_kb":37733,"num_write":62,"num_write_kb":16924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"289'563","reported_seq":1365,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-04-24T02:00:07.104680+0000","last_change":"2024-04-24T02:00:06.132460+0000","last_active":"2024-04-24T02:00:07.104680+0000","last_peered":"2024-04-24T02:00:07.104680+0000","last_clean":"2024-04-24T02:00:07.104680+0000","last_became_active":"2024-04-24T01:55:30.315075+0000","last_became_peered":"2024-04-24T01:55:30.315075+0000","last_unstale":"2024-04-24T02:00:07.104680+0000","last_undegraded":"2024-04-24T02:00:07.104680+0000","last_fullsized":"2024-04-24T02:00:07.104680+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":563,"log_dups_size":0,"ondisk_log_size":563,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":54,"snaptrim_duration":0,"stat_sum":{"num_bytes":11563612,"num_objects":6,"num_object_clones":1,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":257,"num_read_kb":168346,"num_write":252,"num_write_kb":65993,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"289'111","reported_seq":982,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-04-24T02:00:07.122683+0000","last_change":"2024-04-24T02:00:07.122683+0000","last_active":"2024-04-24T02:00:07.122683+0000","last_peered":"2024-04-24T02:00:07.122683+0000","last_clean":"2024-04-24T02:00:07.122683+0000","last_became_active":"2024-04-24T01:55:30.350300+0000","last_became_peered":"2024-04-24T01:55:30.350300+0000","last_unstale":"2024-04-24T02:00:07.122683+0000","last_undegraded":"2024-04-24T02:00:07.122683+0000","last_fullsized":"2024-04-24T02:00:07.122683+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":9,"snaptrim_duration":0,"stat_sum":{"num_bytes":7647350,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":55,"num_read_kb":30805,"num_write":73,"num_write_kb":18631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"264'34","reported_seq":887,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-24T02:00:08.107547+0000","last_change":"2024-04-24T02:00:08.112712+0000","last_active":"2024-04-24T02:00:08.107547+0000","last_peered":"2024-04-24T02:00:08.107547+0000","last_clean":"2024-04-24T02:00:08.107547+0000","last_became_active":"2024-04-24T01:55:30.376813+0000","last_became_peered":"2024-04-24T01:55:30.376813+0000","last_unstale":"2024-04-24T02:00:08.107547+0000","last_undegraded":"2024-04-24T02:00:08.107547+0000","last_fullsized":"2024-04-24T02:00:08.107547+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0,"stat_sum":{"num_bytes":3025359,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":10082,"num_write":27,"num_write_kb":6802,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'97","reported_seq":872,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-04-24T02:00:09.135358+0000","last_change":"2024-04-24T01:55:16.196470+0000","last_active":"2024-04-24T02:00:09.135358+0000","last_peered":"2024-04-24T02:00:09.135358+0000","last_clean":"2024-04-24T02:00:09.135358+0000","last_became_active":"2024-04-24T01:55:16.190464+0000","last_became_peered":"2024-04-24T01:55:16.190464+0000","last_unstale":"2024-04-24T02:00:09.135358+0000","last_undegraded":"2024-04-24T02:00:09.135358+0000","last_fullsized":"2024-04-24T02:00:09.135358+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:15.127115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:15.127115+0000","last_clean_scrub_stamp":"2024-04-24T01:55:15.127115+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":728,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-24T02:00:08.104345+0000","last_change":"2024-04-24T01:55:18.236148+0000","last_active":"2024-04-24T02:00:08.104345+0000","last_peered":"2024-04-24T02:00:08.104345+0000","last_clean":"2024-04-24T02:00:08.104345+0000","last_became_active":"2024-04-24T01:55:18.230154+0000","last_became_peered":"2024-04-24T01:55:18.230154+0000","last_unstale":"2024-04-24T02:00:08.104345+0000","last_undegraded":"2024-04-24T02:00:08.104345+0000","last_fullsized":"2024-04-24T02:00:08.104345+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"291'415","reported_seq":1255,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-24T02:00:08.116956+0000","last_change":"2024-04-24T02:00:08.178565+0000","last_active":"2024-04-24T02:00:08.116956+0000","last_peered":"2024-04-24T02:00:08.116956+0000","last_clean":"2024-04-24T02:00:08.116956+0000","last_became_active":"2024-04-24T01:55:30.316222+0000","last_became_peered":"2024-04-24T01:55:30.316222+0000","last_unstale":"2024-04-24T02:00:08.116956+0000","last_undegraded":"2024-04-24T02:00:08.116956+0000","last_fullsized":"2024-04-24T02:00:08.116956+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":415,"log_dups_size":0,"ondisk_log_size":415,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":33,"snaptrim_duration":0,"stat_sum":{"num_bytes":7378471,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":226,"num_read_kb":163229,"num_write":217,"num_write_kb":60670,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":747,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-04-24T02:00:07.118315+0000","last_change":"2024-04-24T01:55:18.265905+0000","last_active":"2024-04-24T02:00:07.118315+0000","last_peered":"2024-04-24T02:00:07.118315+0000","last_clean":"2024-04-24T02:00:07.118315+0000","last_became_active":"2024-04-24T01:55:18.257020+0000","last_became_peered":"2024-04-24T01:55:18.257020+0000","last_unstale":"2024-04-24T02:00:07.118315+0000","last_undegraded":"2024-04-24T02:00:07.118315+0000","last_fullsized":"2024-04-24T02:00:07.118315+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"289'337","reported_seq":1156,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-04-24T02:00:09.128505+0000","last_change":"2024-04-24T02:00:08.115992+0000","last_active":"2024-04-24T02:00:09.128505+0000","last_peered":"2024-04-24T02:00:09.128505+0000","last_clean":"2024-04-24T02:00:09.128505+0000","last_became_active":"2024-04-24T01:55:30.344129+0000","last_became_peered":"2024-04-24T01:55:30.344129+0000","last_unstale":"2024-04-24T02:00:09.128505+0000","last_undegraded":"2024-04-24T02:00:09.128505+0000","last_fullsized":"2024-04-24T02:00:09.128505+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":337,"log_dups_size":0,"ondisk_log_size":337,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":29,"snaptrim_duration":0,"stat_sum":{"num_bytes":3613604,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":135,"num_read_kb":72541,"num_write":159,"num_write_kb":44990,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":728,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-24T02:00:08.097016+0000","last_change":"2024-04-24T01:55:18.215251+0000","last_active":"2024-04-24T02:00:08.097016+0000","last_peered":"2024-04-24T02:00:08.097016+0000","last_clean":"2024-04-24T02:00:08.097016+0000","last_became_active":"2024-04-24T01:55:18.206605+0000","last_became_peered":"2024-04-24T01:55:18.206605+0000","last_unstale":"2024-04-24T02:00:08.097016+0000","last_undegraded":"2024-04-24T02:00:08.097016+0000","last_fullsized":"2024-04-24T02:00:08.097016+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"289'158","reported_seq":1035,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-04-24T02:00:09.140483+0000","last_change":"2024-04-24T02:00:08.123042+0000","last_active":"2024-04-24T02:00:09.140483+0000","last_peered":"2024-04-24T02:00:09.140483+0000","last_clean":"2024-04-24T02:00:09.140483+0000","last_became_active":"2024-04-24T01:55:30.401366+0000","last_became_peered":"2024-04-24T01:55:30.401366+0000","last_unstale":"2024-04-24T02:00:09.140483+0000","last_undegraded":"2024-04-24T02:00:09.140483+0000","last_fullsized":"2024-04-24T02:00:09.140483+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":158,"log_dups_size":0,"ondisk_log_size":158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":13,"snaptrim_duration":0,"stat_sum":{"num_bytes":6084332,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":138,"num_read_kb":111396,"num_write":61,"num_write_kb":15589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":747,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-04-24T02:00:07.115776+0000","last_change":"2024-04-24T01:55:18.230893+0000","last_active":"2024-04-24T02:00:07.115776+0000","last_peered":"2024-04-24T02:00:07.115776+0000","last_clean":"2024-04-24T02:00:07.115776+0000","last_became_active":"2024-04-24T01:55:18.225188+0000","last_became_peered":"2024-04-24T01:55:18.225188+0000","last_unstale":"2024-04-24T02:00:07.115776+0000","last_undegraded":"2024-04-24T02:00:07.115776+0000","last_fullsized":"2024-04-24T02:00:07.115776+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"292'268","reported_seq":1093,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-04-24T02:00:09.161665+0000","last_change":"2024-04-24T02:00:08.165700+0000","last_active":"2024-04-24T02:00:09.161665+0000","last_peered":"2024-04-24T02:00:09.161665+0000","last_clean":"2024-04-24T02:00:09.161665+0000","last_became_active":"2024-04-24T01:55:30.343318+0000","last_became_peered":"2024-04-24T01:55:30.343318+0000","last_unstale":"2024-04-24T02:00:09.161665+0000","last_undegraded":"2024-04-24T02:00:09.161665+0000","last_fullsized":"2024-04-24T02:00:09.161665+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":268,"log_dups_size":0,"ondisk_log_size":268,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":24,"snaptrim_duration":0,"stat_sum":{"num_bytes":7680997,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":141,"num_read_kb":111499,"num_write":123,"num_write_kb":35132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"291'379","reported_seq":1204,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-24T02:00:08.157938+0000","last_change":"2024-04-24T02:00:08.157938+0000","last_active":"2024-04-24T02:00:08.157938+0000","last_peered":"2024-04-24T02:00:08.157938+0000","last_clean":"2024-04-24T02:00:08.157938+0000","last_became_active":"2024-04-24T01:55:30.345308+0000","last_became_peered":"2024-04-24T01:55:30.345308+0000","last_unstale":"2024-04-24T02:00:08.157938+0000","last_undegraded":"2024-04-24T02:00:08.157938+0000","last_fullsized":"2024-04-24T02:00:08.157938+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":379,"log_dups_size":0,"ondisk_log_size":379,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":30,"snaptrim_duration":0,"stat_sum":{"num_bytes":14252732,"num_objects":6,"num_object_clones":2,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":186,"num_read_kb":114673,"num_write":188,"num_write_kb":50556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":721,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-04-24T02:00:09.109311+0000","last_change":"2024-04-24T01:55:18.224238+0000","last_active":"2024-04-24T02:00:09.109311+0000","last_peered":"2024-04-24T02:00:09.109311+0000","last_clean":"2024-04-24T02:00:09.109311+0000","last_became_active":"2024-04-24T01:55:18.219131+0000","last_became_peered":"2024-04-24T01:55:18.219131+0000","last_unstale":"2024-04-24T02:00:09.109311+0000","last_undegraded":"2024-04-24T02:00:09.109311+0000","last_fullsized":"2024-04-24T02:00:09.109311+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"291'79","reported_seq":926,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-04-24T02:00:09.115271+0000","last_change":"2024-04-24T02:00:08.167396+0000","last_active":"2024-04-24T02:00:09.115271+0000","last_peered":"2024-04-24T02:00:09.115271+0000","last_clean":"2024-04-24T02:00:09.115271+0000","last_became_active":"2024-04-24T01:55:30.319958+0000","last_became_peered":"2024-04-24T01:55:30.319958+0000","last_unstale":"2024-04-24T02:00:09.115271+0000","last_undegraded":"2024-04-24T02:00:09.115271+0000","last_fullsized":"2024-04-24T02:00:09.115271+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0,"stat_sum":{"num_bytes":3882516,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":41,"num_read_kb":36474,"num_write":51,"num_write_kb":14818,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":739,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-04-24T02:00:09.142176+0000","last_change":"2024-04-24T01:55:18.235763+0000","last_active":"2024-04-24T02:00:09.142176+0000","last_peered":"2024-04-24T02:00:09.142176+0000","last_clean":"2024-04-24T02:00:09.142176+0000","last_became_active":"2024-04-24T01:55:18.229175+0000","last_became_peered":"2024-04-24T01:55:18.229175+0000","last_unstale":"2024-04-24T02:00:09.142176+0000","last_undegraded":"2024-04-24T02:00:09.142176+0000","last_fullsized":"2024-04-24T02:00:09.142176+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"281'149","reported_seq":994,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-04-24T02:00:09.131134+0000","last_change":"2024-04-24T02:00:08.116309+0000","last_active":"2024-04-24T02:00:09.131134+0000","last_peered":"2024-04-24T02:00:09.131134+0000","last_clean":"2024-04-24T02:00:09.131134+0000","last_became_active":"2024-04-24T01:55:30.372690+0000","last_became_peered":"2024-04-24T01:55:30.372690+0000","last_unstale":"2024-04-24T02:00:09.131134+0000","last_undegraded":"2024-04-24T02:00:09.131134+0000","last_fullsized":"2024-04-24T02:00:09.131134+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":13,"snaptrim_duration":0,"stat_sum":{"num_bytes":6857454,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":62,"num_read_kb":39376,"num_write":69,"num_write_kb":19977,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":721,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-04-24T02:00:09.110469+0000","last_change":"2024-04-24T01:55:18.278351+0000","last_active":"2024-04-24T02:00:09.110469+0000","last_peered":"2024-04-24T02:00:09.110469+0000","last_clean":"2024-04-24T02:00:09.110469+0000","last_became_active":"2024-04-24T01:55:18.273668+0000","last_became_peered":"2024-04-24T01:55:18.273668+0000","last_unstale":"2024-04-24T02:00:09.110469+0000","last_undegraded":"2024-04-24T02:00:09.110469+0000","last_fullsized":"2024-04-24T02:00:09.110469+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"291'327","reported_seq":1179,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-04-24T02:00:09.111384+0000","last_change":"2024-04-24T02:00:08.123084+0000","last_active":"2024-04-24T02:00:09.111384+0000","last_peered":"2024-04-24T02:00:09.111384+0000","last_clean":"2024-04-24T02:00:09.111384+0000","last_became_active":"2024-04-24T01:55:30.363123+0000","last_became_peered":"2024-04-24T01:55:30.363123+0000","last_unstale":"2024-04-24T02:00:09.111384+0000","last_undegraded":"2024-04-24T02:00:09.111384+0000","last_fullsized":"2024-04-24T02:00:09.111384+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":327,"log_dups_size":0,"ondisk_log_size":327,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":35,"snaptrim_duration":0,"stat_sum":{"num_bytes":17849529,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":215,"num_read_kb":175874,"num_write":177,"num_write_kb":48260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":728,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-24T02:00:08.108670+0000","last_change":"2024-04-24T01:55:18.241117+0000","last_active":"2024-04-24T02:00:08.108670+0000","last_peered":"2024-04-24T02:00:08.108670+0000","last_clean":"2024-04-24T02:00:08.108670+0000","last_became_active":"2024-04-24T01:55:18.234313+0000","last_became_peered":"2024-04-24T01:55:18.234313+0000","last_unstale":"2024-04-24T02:00:08.108670+0000","last_undegraded":"2024-04-24T02:00:08.108670+0000","last_fullsized":"2024-04-24T02:00:08.108670+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":118778468,"num_objects":49,"num_object_clones":11,"num_object_copies":147,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":2143,"num_read_kb":1530942,"num_write":1986,"num_write_kb":544042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":248877056,"data_stored":265832857,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3976,"ondisk_log_size":3976,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":1}],"osd_stats":[{"osd":3,"up_from":10,"seq":42949673254,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":102036,"kb_used_data":66892,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94269804,"statfs":{"total":96636764160,"available":96532279296,"internally_reserved":0,"allocated":68497408,"data_stored":72212874,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673254,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":96792,"kb_used_data":60208,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94275048,"statfs":{"total":96636764160,"available":96537649152,"internally_reserved":0,"allocated":61652992,"data_stored":64894925,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705959,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":84716,"kb_used_data":49572,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94287124,"statfs":{"total":96636764160,"available":96550014976,"internally_reserved":0,"allocated":50761728,"data_stored":52736552,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705958,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":106532,"kb_used_data":71388,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94265308,"statfs":{"total":96636764160,"available":96527675392,"internally_reserved":0,"allocated":73101312,"data_stored":77388167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":50073600,"data_stored":52842748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":70721536,"data_stored":76348561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":60956672,"data_stored":64997444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":67125248,"data_stored":71644104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T02:00:12.466 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T02:00:12.571 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:12.567+0000 7fa06a79d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:12.575 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:12.574+0000 7fa06a79d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:12.761 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T02:00:12.762 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-24T02:00:12.774 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":435,"stamp":"2024-04-24T02:00:12.058449+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":119368836,"num_objects":51,"num_object_clones":11,"num_object_copies":153,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":2246,"num_read_kb":1531033,"num_write":2123,"num_write_kb":545936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4073,"ondisk_log_size":4073,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":390076,"kb_used_data":248060,"kb_used_omap":0,"kb_used_meta":140544,"kb_avail":377097284,"statfs":{"total":386547056640,"available":386147618816,"internally_reserved":0,"allocated":254013440,"data_stored":267232518,"data_compressed":89889,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":143917056},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5815222,"num_objects":-2,"num_object_clones":-3,"num_object_copies":-6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":31,"num_read_kb":25382,"num_write":37,"num_write_kb":11428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.975360"},"pg_stats":[{"pgid":"3.f","version":"289'214","reported_seq":1096,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-04-24T02:00:07.116472+0000","last_change":"2024-04-24T02:00:07.116472+0000","last_active":"2024-04-24T02:00:07.116472+0000","last_peered":"2024-04-24T02:00:07.116472+0000","last_clean":"2024-04-24T02:00:07.116472+0000","last_became_active":"2024-04-24T01:55:30.386726+0000","last_became_peered":"2024-04-24T01:55:30.386726+0000","last_unstale":"2024-04-24T02:00:07.116472+0000","last_undegraded":"2024-04-24T02:00:07.116472+0000","last_fullsized":"2024-04-24T02:00:07.116472+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":214,"log_dups_size":0,"ondisk_log_size":214,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":20,"snaptrim_duration":0,"stat_sum":{"num_bytes":7059138,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":163,"num_read_kb":147337,"num_write":103,"num_write_kb":30109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"291'324","reported_seq":1154,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-04-24T02:00:09.100093+0000","last_change":"2024-04-24T02:00:08.152477+0000","last_active":"2024-04-24T02:00:09.100093+0000","last_peered":"2024-04-24T02:00:09.100093+0000","last_clean":"2024-04-24T02:00:09.100093+0000","last_became_active":"2024-04-24T01:55:30.321866+0000","last_became_peered":"2024-04-24T01:55:30.321866+0000","last_unstale":"2024-04-24T02:00:09.100093+0000","last_undegraded":"2024-04-24T02:00:09.100093+0000","last_fullsized":"2024-04-24T02:00:09.100093+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":324,"log_dups_size":0,"ondisk_log_size":324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":30,"snaptrim_duration":0,"stat_sum":{"num_bytes":9916427,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":175,"num_read_kb":116839,"num_write":161,"num_write_kb":44263,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"264'229","reported_seq":1071,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-24T02:00:08.096053+0000","last_change":"2024-04-24T02:00:08.102657+0000","last_active":"2024-04-24T02:00:08.096053+0000","last_peered":"2024-04-24T02:00:08.096053+0000","last_clean":"2024-04-24T02:00:08.096053+0000","last_became_active":"2024-04-24T01:55:30.320186+0000","last_became_peered":"2024-04-24T01:55:30.320186+0000","last_unstale":"2024-04-24T02:00:08.096053+0000","last_undegraded":"2024-04-24T02:00:08.096053+0000","last_fullsized":"2024-04-24T02:00:08.096053+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":229,"log_dups_size":0,"ondisk_log_size":229,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":24,"snaptrim_duration":0,"stat_sum":{"num_bytes":5159914,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":125,"num_read_kb":85296,"num_write":124,"num_write_kb":34018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"289'282","reported_seq":1137,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-04-24T02:00:07.117089+0000","last_change":"2024-04-24T02:00:07.117089+0000","last_active":"2024-04-24T02:00:07.117089+0000","last_peered":"2024-04-24T02:00:07.117089+0000","last_clean":"2024-04-24T02:00:07.117089+0000","last_became_active":"2024-04-24T01:55:30.288081+0000","last_became_peered":"2024-04-24T01:55:30.288081+0000","last_unstale":"2024-04-24T02:00:07.117089+0000","last_undegraded":"2024-04-24T02:00:07.117089+0000","last_fullsized":"2024-04-24T02:00:07.117089+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":282,"log_dups_size":0,"ondisk_log_size":282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":28,"snaptrim_duration":0,"stat_sum":{"num_bytes":6807033,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":150,"num_read_kb":109442,"num_write":139,"num_write_kb":37310,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"277'107","reported_seq":969,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-04-24T02:00:09.144222+0000","last_change":"2024-04-24T02:00:08.124445+0000","last_active":"2024-04-24T02:00:09.144222+0000","last_peered":"2024-04-24T02:00:09.144222+0000","last_clean":"2024-04-24T02:00:09.144222+0000","last_became_active":"2024-04-24T01:55:30.333815+0000","last_became_peered":"2024-04-24T01:55:30.333815+0000","last_unstale":"2024-04-24T02:00:09.144222+0000","last_undegraded":"2024-04-24T02:00:09.144222+0000","last_fullsized":"2024-04-24T02:00:09.144222+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":12,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":59,"num_read_kb":37733,"num_write":62,"num_write_kb":16924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"289'563","reported_seq":1365,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-04-24T02:00:07.104680+0000","last_change":"2024-04-24T02:00:06.132460+0000","last_active":"2024-04-24T02:00:07.104680+0000","last_peered":"2024-04-24T02:00:07.104680+0000","last_clean":"2024-04-24T02:00:07.104680+0000","last_became_active":"2024-04-24T01:55:30.315075+0000","last_became_peered":"2024-04-24T01:55:30.315075+0000","last_unstale":"2024-04-24T02:00:07.104680+0000","last_undegraded":"2024-04-24T02:00:07.104680+0000","last_fullsized":"2024-04-24T02:00:07.104680+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":563,"log_dups_size":0,"ondisk_log_size":563,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":54,"snaptrim_duration":0,"stat_sum":{"num_bytes":11563612,"num_objects":6,"num_object_clones":1,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":257,"num_read_kb":168346,"num_write":252,"num_write_kb":65993,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"289'111","reported_seq":982,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-04-24T02:00:07.122683+0000","last_change":"2024-04-24T02:00:07.122683+0000","last_active":"2024-04-24T02:00:07.122683+0000","last_peered":"2024-04-24T02:00:07.122683+0000","last_clean":"2024-04-24T02:00:07.122683+0000","last_became_active":"2024-04-24T01:55:30.350300+0000","last_became_peered":"2024-04-24T01:55:30.350300+0000","last_unstale":"2024-04-24T02:00:07.122683+0000","last_undegraded":"2024-04-24T02:00:07.122683+0000","last_fullsized":"2024-04-24T02:00:07.122683+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":9,"snaptrim_duration":0,"stat_sum":{"num_bytes":7647350,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":55,"num_read_kb":30805,"num_write":73,"num_write_kb":18631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"264'34","reported_seq":887,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-24T02:00:08.107547+0000","last_change":"2024-04-24T02:00:08.112712+0000","last_active":"2024-04-24T02:00:08.107547+0000","last_peered":"2024-04-24T02:00:08.107547+0000","last_clean":"2024-04-24T02:00:08.107547+0000","last_became_active":"2024-04-24T01:55:30.376813+0000","last_became_peered":"2024-04-24T01:55:30.376813+0000","last_unstale":"2024-04-24T02:00:08.107547+0000","last_undegraded":"2024-04-24T02:00:08.107547+0000","last_fullsized":"2024-04-24T02:00:08.107547+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0,"stat_sum":{"num_bytes":3025359,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":10082,"num_write":27,"num_write_kb":6802,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'97","reported_seq":872,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-04-24T02:00:09.135358+0000","last_change":"2024-04-24T01:55:16.196470+0000","last_active":"2024-04-24T02:00:09.135358+0000","last_peered":"2024-04-24T02:00:09.135358+0000","last_clean":"2024-04-24T02:00:09.135358+0000","last_became_active":"2024-04-24T01:55:16.190464+0000","last_became_peered":"2024-04-24T01:55:16.190464+0000","last_unstale":"2024-04-24T02:00:09.135358+0000","last_undegraded":"2024-04-24T02:00:09.135358+0000","last_fullsized":"2024-04-24T02:00:09.135358+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:15.127115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:15.127115+0000","last_clean_scrub_stamp":"2024-04-24T01:55:15.127115+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":728,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-24T02:00:08.104345+0000","last_change":"2024-04-24T01:55:18.236148+0000","last_active":"2024-04-24T02:00:08.104345+0000","last_peered":"2024-04-24T02:00:08.104345+0000","last_clean":"2024-04-24T02:00:08.104345+0000","last_became_active":"2024-04-24T01:55:18.230154+0000","last_became_peered":"2024-04-24T01:55:18.230154+0000","last_unstale":"2024-04-24T02:00:08.104345+0000","last_undegraded":"2024-04-24T02:00:08.104345+0000","last_fullsized":"2024-04-24T02:00:08.104345+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"291'415","reported_seq":1255,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-24T02:00:08.116956+0000","last_change":"2024-04-24T02:00:08.178565+0000","last_active":"2024-04-24T02:00:08.116956+0000","last_peered":"2024-04-24T02:00:08.116956+0000","last_clean":"2024-04-24T02:00:08.116956+0000","last_became_active":"2024-04-24T01:55:30.316222+0000","last_became_peered":"2024-04-24T01:55:30.316222+0000","last_unstale":"2024-04-24T02:00:08.116956+0000","last_undegraded":"2024-04-24T02:00:08.116956+0000","last_fullsized":"2024-04-24T02:00:08.116956+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":415,"log_dups_size":0,"ondisk_log_size":415,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":33,"snaptrim_duration":0,"stat_sum":{"num_bytes":7378471,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":226,"num_read_kb":163229,"num_write":217,"num_write_kb":60670,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":747,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-04-24T02:00:07.118315+0000","last_change":"2024-04-24T01:55:18.265905+0000","last_active":"2024-04-24T02:00:07.118315+0000","last_peered":"2024-04-24T02:00:07.118315+0000","last_clean":"2024-04-24T02:00:07.118315+0000","last_became_active":"2024-04-24T01:55:18.257020+0000","last_became_peered":"2024-04-24T01:55:18.257020+0000","last_unstale":"2024-04-24T02:00:07.118315+0000","last_undegraded":"2024-04-24T02:00:07.118315+0000","last_fullsized":"2024-04-24T02:00:07.118315+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"289'337","reported_seq":1156,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-04-24T02:00:09.128505+0000","last_change":"2024-04-24T02:00:08.115992+0000","last_active":"2024-04-24T02:00:09.128505+0000","last_peered":"2024-04-24T02:00:09.128505+0000","last_clean":"2024-04-24T02:00:09.128505+0000","last_became_active":"2024-04-24T01:55:30.344129+0000","last_became_peered":"2024-04-24T01:55:30.344129+0000","last_unstale":"2024-04-24T02:00:09.128505+0000","last_undegraded":"2024-04-24T02:00:09.128505+0000","last_fullsized":"2024-04-24T02:00:09.128505+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":337,"log_dups_size":0,"ondisk_log_size":337,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":29,"snaptrim_duration":0,"stat_sum":{"num_bytes":3613604,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":135,"num_read_kb":72541,"num_write":159,"num_write_kb":44990,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":728,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-24T02:00:08.097016+0000","last_change":"2024-04-24T01:55:18.215251+0000","last_active":"2024-04-24T02:00:08.097016+0000","last_peered":"2024-04-24T02:00:08.097016+0000","last_clean":"2024-04-24T02:00:08.097016+0000","last_became_active":"2024-04-24T01:55:18.206605+0000","last_became_peered":"2024-04-24T01:55:18.206605+0000","last_unstale":"2024-04-24T02:00:08.097016+0000","last_undegraded":"2024-04-24T02:00:08.097016+0000","last_fullsized":"2024-04-24T02:00:08.097016+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"289'158","reported_seq":1035,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-04-24T02:00:09.140483+0000","last_change":"2024-04-24T02:00:08.123042+0000","last_active":"2024-04-24T02:00:09.140483+0000","last_peered":"2024-04-24T02:00:09.140483+0000","last_clean":"2024-04-24T02:00:09.140483+0000","last_became_active":"2024-04-24T01:55:30.401366+0000","last_became_peered":"2024-04-24T01:55:30.401366+0000","last_unstale":"2024-04-24T02:00:09.140483+0000","last_undegraded":"2024-04-24T02:00:09.140483+0000","last_fullsized":"2024-04-24T02:00:09.140483+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":158,"log_dups_size":0,"ondisk_log_size":158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":13,"snaptrim_duration":0,"stat_sum":{"num_bytes":6084332,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":138,"num_read_kb":111396,"num_write":61,"num_write_kb":15589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":747,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-04-24T02:00:07.115776+0000","last_change":"2024-04-24T01:55:18.230893+0000","last_active":"2024-04-24T02:00:07.115776+0000","last_peered":"2024-04-24T02:00:07.115776+0000","last_clean":"2024-04-24T02:00:07.115776+0000","last_became_active":"2024-04-24T01:55:18.225188+0000","last_became_peered":"2024-04-24T01:55:18.225188+0000","last_unstale":"2024-04-24T02:00:07.115776+0000","last_undegraded":"2024-04-24T02:00:07.115776+0000","last_fullsized":"2024-04-24T02:00:07.115776+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"292'268","reported_seq":1093,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-04-24T02:00:09.161665+0000","last_change":"2024-04-24T02:00:08.165700+0000","last_active":"2024-04-24T02:00:09.161665+0000","last_peered":"2024-04-24T02:00:09.161665+0000","last_clean":"2024-04-24T02:00:09.161665+0000","last_became_active":"2024-04-24T01:55:30.343318+0000","last_became_peered":"2024-04-24T01:55:30.343318+0000","last_unstale":"2024-04-24T02:00:09.161665+0000","last_undegraded":"2024-04-24T02:00:09.161665+0000","last_fullsized":"2024-04-24T02:00:09.161665+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":268,"log_dups_size":0,"ondisk_log_size":268,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":24,"snaptrim_duration":0,"stat_sum":{"num_bytes":7680997,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":141,"num_read_kb":111499,"num_write":123,"num_write_kb":35132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"291'379","reported_seq":1204,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-24T02:00:08.157938+0000","last_change":"2024-04-24T02:00:08.157938+0000","last_active":"2024-04-24T02:00:08.157938+0000","last_peered":"2024-04-24T02:00:08.157938+0000","last_clean":"2024-04-24T02:00:08.157938+0000","last_became_active":"2024-04-24T01:55:30.345308+0000","last_became_peered":"2024-04-24T01:55:30.345308+0000","last_unstale":"2024-04-24T02:00:08.157938+0000","last_undegraded":"2024-04-24T02:00:08.157938+0000","last_fullsized":"2024-04-24T02:00:08.157938+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":379,"log_dups_size":0,"ondisk_log_size":379,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":30,"snaptrim_duration":0,"stat_sum":{"num_bytes":14252732,"num_objects":6,"num_object_clones":2,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":186,"num_read_kb":114673,"num_write":188,"num_write_kb":50556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":721,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-04-24T02:00:09.109311+0000","last_change":"2024-04-24T01:55:18.224238+0000","last_active":"2024-04-24T02:00:09.109311+0000","last_peered":"2024-04-24T02:00:09.109311+0000","last_clean":"2024-04-24T02:00:09.109311+0000","last_became_active":"2024-04-24T01:55:18.219131+0000","last_became_peered":"2024-04-24T01:55:18.219131+0000","last_unstale":"2024-04-24T02:00:09.109311+0000","last_undegraded":"2024-04-24T02:00:09.109311+0000","last_fullsized":"2024-04-24T02:00:09.109311+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"291'79","reported_seq":926,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-04-24T02:00:09.115271+0000","last_change":"2024-04-24T02:00:08.167396+0000","last_active":"2024-04-24T02:00:09.115271+0000","last_peered":"2024-04-24T02:00:09.115271+0000","last_clean":"2024-04-24T02:00:09.115271+0000","last_became_active":"2024-04-24T01:55:30.319958+0000","last_became_peered":"2024-04-24T01:55:30.319958+0000","last_unstale":"2024-04-24T02:00:09.115271+0000","last_undegraded":"2024-04-24T02:00:09.115271+0000","last_fullsized":"2024-04-24T02:00:09.115271+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0,"stat_sum":{"num_bytes":3882516,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":41,"num_read_kb":36474,"num_write":51,"num_write_kb":14818,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":739,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-04-24T02:00:09.142176+0000","last_change":"2024-04-24T01:55:18.235763+0000","last_active":"2024-04-24T02:00:09.142176+0000","last_peered":"2024-04-24T02:00:09.142176+0000","last_clean":"2024-04-24T02:00:09.142176+0000","last_became_active":"2024-04-24T01:55:18.229175+0000","last_became_peered":"2024-04-24T01:55:18.229175+0000","last_unstale":"2024-04-24T02:00:09.142176+0000","last_undegraded":"2024-04-24T02:00:09.142176+0000","last_fullsized":"2024-04-24T02:00:09.142176+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"281'149","reported_seq":994,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-04-24T02:00:09.131134+0000","last_change":"2024-04-24T02:00:08.116309+0000","last_active":"2024-04-24T02:00:09.131134+0000","last_peered":"2024-04-24T02:00:09.131134+0000","last_clean":"2024-04-24T02:00:09.131134+0000","last_became_active":"2024-04-24T01:55:30.372690+0000","last_became_peered":"2024-04-24T01:55:30.372690+0000","last_unstale":"2024-04-24T02:00:09.131134+0000","last_undegraded":"2024-04-24T02:00:09.131134+0000","last_fullsized":"2024-04-24T02:00:09.131134+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":13,"snaptrim_duration":0,"stat_sum":{"num_bytes":6857454,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":62,"num_read_kb":39376,"num_write":69,"num_write_kb":19977,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":721,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-04-24T02:00:09.110469+0000","last_change":"2024-04-24T01:55:18.278351+0000","last_active":"2024-04-24T02:00:09.110469+0000","last_peered":"2024-04-24T02:00:09.110469+0000","last_clean":"2024-04-24T02:00:09.110469+0000","last_became_active":"2024-04-24T01:55:18.273668+0000","last_became_peered":"2024-04-24T01:55:18.273668+0000","last_unstale":"2024-04-24T02:00:09.110469+0000","last_undegraded":"2024-04-24T02:00:09.110469+0000","last_fullsized":"2024-04-24T02:00:09.110469+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"291'327","reported_seq":1179,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-04-24T02:00:09.111384+0000","last_change":"2024-04-24T02:00:08.123084+0000","last_active":"2024-04-24T02:00:09.111384+0000","last_peered":"2024-04-24T02:00:09.111384+0000","last_clean":"2024-04-24T02:00:09.111384+0000","last_became_active":"2024-04-24T01:55:30.363123+0000","last_became_peered":"2024-04-24T01:55:30.363123+0000","last_unstale":"2024-04-24T02:00:09.111384+0000","last_undegraded":"2024-04-24T02:00:09.111384+0000","last_fullsized":"2024-04-24T02:00:09.111384+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":327,"log_dups_size":0,"ondisk_log_size":327,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":35,"snaptrim_duration":0,"stat_sum":{"num_bytes":17849529,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":215,"num_read_kb":175874,"num_write":177,"num_write_kb":48260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":728,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-04-24T02:00:08.108670+0000","last_change":"2024-04-24T01:55:18.241117+0000","last_active":"2024-04-24T02:00:08.108670+0000","last_peered":"2024-04-24T02:00:08.108670+0000","last_clean":"2024-04-24T02:00:08.108670+0000","last_became_active":"2024-04-24T01:55:18.234313+0000","last_became_peered":"2024-04-24T01:55:18.234313+0000","last_unstale":"2024-04-24T02:00:08.108670+0000","last_undegraded":"2024-04-24T02:00:08.108670+0000","last_fullsized":"2024-04-24T02:00:08.108670+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":118778468,"num_objects":49,"num_object_clones":11,"num_object_copies":147,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":2143,"num_read_kb":1530942,"num_write":1986,"num_write_kb":544042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":248877056,"data_stored":265832857,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3976,"ondisk_log_size":3976,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":1}],"osd_stats":[{"osd":3,"up_from":10,"seq":42949673254,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":102036,"kb_used_data":66892,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94269804,"statfs":{"total":96636764160,"available":96532279296,"internally_reserved":0,"allocated":68497408,"data_stored":72212874,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673254,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":96792,"kb_used_data":60208,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94275048,"statfs":{"total":96636764160,"available":96537649152,"internally_reserved":0,"allocated":61652992,"data_stored":64894925,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705959,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":84716,"kb_used_data":49572,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94287124,"statfs":{"total":96636764160,"available":96550014976,"internally_reserved":0,"allocated":50761728,"data_stored":52736552,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705958,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":106532,"kb_used_data":71388,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94265308,"statfs":{"total":96636764160,"available":96527675392,"internally_reserved":0,"allocated":73101312,"data_stored":77388167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":50073600,"data_stored":52842748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":70721536,"data_stored":76348561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":60956672,"data_stored":64997444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":67125248,"data_stored":71644104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T02:00:12.775 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-04-24T02:00:12.775 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-04-24T02:00:12.775 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-04-24T02:00:12.776 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-04-24T02:00:12.884 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:12.878+0000 7f507b06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:12.887 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:12.887+0000 7f507b06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:12.902 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T02:00:12.902 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T02:00:12.902 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T02:00:12.902 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T02:00:12.902 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T02:00:12.902 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T02:00:12.903 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T02:00:12.903 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T02:00:12.903 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T02:00:12.903 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T02:00:12.903 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T02:00:12.903 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T02:00:12.905 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T02:00:12.905 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T02:00:12.906 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-04-24T02:00:13.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:13.007+0000 7f325842d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:13.016 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:13.015+0000 7f325842d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:13.030 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T02:00:13.030 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T02:00:13.030 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T02:00:13.030 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T02:00:13.031 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T02:00:13.031 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T02:00:13.031 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T02:00:13.031 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T02:00:13.031 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T02:00:13.031 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T02:00:13.031 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T02:00:13.031 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T02:00:13.033 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T02:00:13.034 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T02:00:13.034 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-04-24T02:00:13.100 INFO:tasks.rados.rados.0.smithi079.stdout:1231: done (2 left) 2024-04-24T02:00:13.100 INFO:tasks.rados.rados.0.smithi079.stdout:1232: done (1 left) 2024-04-24T02:00:13.100 INFO:tasks.rados.rados.0.smithi079.stdout:1233: done (0 left) 2024-04-24T02:00:13.100 INFO:tasks.rados.rados.0.smithi079.stdout:1234: write oid 32 current snap is 141 2024-04-24T02:00:13.100 INFO:tasks.rados.rados.0.smithi079.stdout:1234: seq_num 603 ranges {534621=567034,1636289=498332} 2024-04-24T02:00:13.107 INFO:tasks.rados.rados.0.smithi079.stdout:1234: writing smithi07929762-32 from 534621 to 1101655 tid 1 2024-04-24T02:00:13.114 INFO:tasks.rados.rados.0.smithi079.stdout:1234: writing smithi07929762-32 from 1636289 to 2134621 tid 2 2024-04-24T02:00:13.114 INFO:tasks.rados.rados.0.smithi079.stdout:1235: delete oid 34 current snap is 141 2024-04-24T02:00:13.157 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:13.150+0000 7fdb3c66d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:13.163 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:13.162+0000 7fdb3c66d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:13.204 INFO:tasks.rados.rados.0.smithi079.stdout:1234: finishing write tid 1 to smithi07929762-32 2024-04-24T02:00:13.221 INFO:tasks.rados.rados.0.smithi079.stdout:1234: finishing write tid 2 to smithi07929762-32 2024-04-24T02:00:13.222 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T02:00:13.222 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T02:00:13.222 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T02:00:13.222 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T02:00:13.222 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T02:00:13.222 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T02:00:13.222 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T02:00:13.222 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T02:00:13.222 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T02:00:13.223 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T02:00:13.223 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T02:00:13.223 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T02:00:13.225 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T02:00:13.225 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T02:00:13.225 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-04-24T02:00:13.239 INFO:tasks.rados.rados.0.smithi079.stdout:1234: finishing write tid 3 to smithi07929762-32 2024-04-24T02:00:13.241 INFO:tasks.rados.rados.0.smithi079.stdout:1235: done (1 left) 2024-04-24T02:00:13.241 INFO:tasks.rados.rados.0.smithi079.stdout:1236: read oid 20 snap -1 2024-04-24T02:00:13.241 INFO:tasks.rados.rados.0.smithi079.stdout:1236: expect (ObjNum 565 snap 130 seq_num 565) 2024-04-24T02:00:13.241 INFO:tasks.rados.rados.0.smithi079.stdout:1237: delete oid 36 current snap is 141 2024-04-24T02:00:13.243 INFO:tasks.rados.rados.0.smithi079.stdout:1234: finishing write tid 4 to smithi07929762-32 2024-04-24T02:00:13.243 INFO:tasks.rados.rados.0.smithi079.stdout:1234: oid 32 updating version 0 to 220 2024-04-24T02:00:13.243 INFO:tasks.rados.rados.0.smithi079.stdout:1234: oid 32 version 220 is already newer than 220 2024-04-24T02:00:13.243 INFO:tasks.rados.rados.0.smithi079.stdout:1234: oid 32 version 220 is already newer than 220 2024-04-24T02:00:13.243 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 32 v 220 (ObjNum 603 snap 141 seq_num 603) dirty exists 2024-04-24T02:00:13.243 INFO:tasks.rados.rados.0.smithi079.stdout:1234: left oid 32 (ObjNum 603 snap 141 seq_num 603) 2024-04-24T02:00:13.245 INFO:tasks.rados.rados.0.smithi079.stdout:1234: done (2 left) 2024-04-24T02:00:13.245 INFO:tasks.rados.rados.0.smithi079.stdout:1237: done (1 left) 2024-04-24T02:00:13.245 INFO:tasks.rados.rados.0.smithi079.stdout:1238: snap_create 2024-04-24T02:00:13.263 INFO:tasks.rados.rados.0.smithi079.stdout:1236: expect (ObjNum 565 snap 130 seq_num 565) 2024-04-24T02:00:13.364 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:13.358+0000 7f2b64b5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:13.367 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:13.367+0000 7f2b64b5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:13.383 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T02:00:13.383 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T02:00:13.383 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T02:00:13.384 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T02:00:13.384 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T02:00:13.384 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T02:00:13.384 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T02:00:13.384 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T02:00:13.384 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T02:00:13.384 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T02:00:13.384 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T02:00:13.384 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T02:00:13.387 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T02:00:13.388 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T02:00:13.438 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T02:00:13.543 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:13.541+0000 7f20f746d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:13.547 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:13.546+0000 7f20f746d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:14.103 INFO:tasks.rados.rados.0.smithi079.stdout:1236: done (1 left) 2024-04-24T02:00:14.103 INFO:tasks.rados.rados.0.smithi079.stdout:1238: done (0 left) 2024-04-24T02:00:14.103 INFO:tasks.rados.rados.0.smithi079.stdout:1239: delete oid 15 current snap is 142 2024-04-24T02:00:14.169 INFO:tasks.rados.rados.0.smithi079.stdout:1239: done (0 left) 2024-04-24T02:00:14.170 INFO:tasks.rados.rados.0.smithi079.stdout:1240: read oid 32 snap -1 2024-04-24T02:00:14.170 INFO:tasks.rados.rados.0.smithi079.stdout:1240: expect (ObjNum 603 snap 141 seq_num 603) 2024-04-24T02:00:14.170 INFO:tasks.rados.rados.0.smithi079.stdout:1241: read oid 49 snap -1 2024-04-24T02:00:14.170 INFO:tasks.rados.rados.0.smithi079.stdout:1241: expect (ObjNum 499 snap 112 seq_num 499) 2024-04-24T02:00:14.170 INFO:tasks.rados.rados.0.smithi079.stdout:1242: read oid 37 snap 136 2024-04-24T02:00:14.170 INFO:tasks.rados.rados.0.smithi079.stdout:1242: expect deleted 2024-04-24T02:00:14.170 INFO:tasks.rados.rados.0.smithi079.stdout:1243: write (excl) oid 28 current snap is 142 2024-04-24T02:00:14.171 INFO:tasks.rados.rados.0.smithi079.stdout:1243: seq_num 607 ranges {601849=468030,1753864=447985} 2024-04-24T02:00:14.174 INFO:tasks.rados.rados.0.smithi079.stdout:1243: writing smithi07929762-28 from 601849 to 1069879 tid 1 2024-04-24T02:00:14.181 INFO:tasks.rados.rados.0.smithi079.stdout:1243: writing smithi07929762-28 from 1753864 to 2201849 tid 2 2024-04-24T02:00:14.181 INFO:tasks.rados.rados.0.smithi079.stdout:1244: read oid 34 snap -1 2024-04-24T02:00:14.181 INFO:tasks.rados.rados.0.smithi079.stdout:1244: expect deleted 2024-04-24T02:00:14.181 INFO:tasks.rados.rados.0.smithi079.stdout:1245: write oid 40 current snap is 142 2024-04-24T02:00:14.181 INFO:tasks.rados.rados.0.smithi079.stdout:1245: seq_num 608 ranges {618656=543279,1812346=406310} 2024-04-24T02:00:14.189 INFO:tasks.rados.rados.0.smithi079.stdout:1245: writing smithi07929762-40 from 618656 to 1161935 tid 1 2024-04-24T02:00:14.197 INFO:tasks.rados.rados.0.smithi079.stdout:1245: writing smithi07929762-40 from 1812346 to 2218656 tid 2 2024-04-24T02:00:14.197 INFO:tasks.rados.rados.0.smithi079.stdout:1246: write (excl) oid 48 current snap is 142 2024-04-24T02:00:14.197 INFO:tasks.rados.rados.0.smithi079.stdout:1246: seq_num 609 ranges {635463=734881,1987181=248282} 2024-04-24T02:00:14.204 INFO:tasks.rados.rados.0.smithi079.stdout:1246: writing smithi07929762-48 from 635463 to 1370344 tid 1 2024-04-24T02:00:14.211 INFO:tasks.rados.rados.0.smithi079.stdout:1246: writing smithi07929762-48 from 1987181 to 2235463 tid 2 2024-04-24T02:00:14.211 INFO:tasks.rados.rados.0.smithi079.stdout:1247: snap_remove snap 141 2024-04-24T02:00:15.110 INFO:tasks.rados.rados.0.smithi079.stdout:1247: done (7 left) 2024-04-24T02:00:15.110 INFO:tasks.rados.rados.0.smithi079.stdout:1248: read oid 15 snap -1 2024-04-24T02:00:15.110 INFO:tasks.rados.rados.0.smithi079.stdout:1248: expect deleted 2024-04-24T02:00:15.110 INFO:tasks.rados.rados.0.smithi079.stdout:1249: write oid 27 current snap is 142 2024-04-24T02:00:15.110 INFO:tasks.rados.rados.0.smithi079.stdout:1243: finishing write tid 1 to smithi07929762-28 2024-04-24T02:00:15.111 INFO:tasks.rados.rados.0.smithi079.stdout:1249: seq_num 610 ranges {652270=410130,1529310=493867} 2024-04-24T02:00:15.116 INFO:tasks.rados.rados.0.smithi079.stdout:1249: writing smithi07929762-27 from 652270 to 1062400 tid 1 2024-04-24T02:00:15.121 INFO:tasks.rados.rados.0.smithi079.stdout:1249: writing smithi07929762-27 from 1529310 to 2023177 tid 2 2024-04-24T02:00:15.121 INFO:tasks.rados.rados.0.smithi079.stdout:1250: read oid 21 snap -1 2024-04-24T02:00:15.121 INFO:tasks.rados.rados.0.smithi079.stdout:1250: expect (ObjNum 500 snap 112 seq_num 500) 2024-04-24T02:00:15.121 INFO:tasks.rados.rados.0.smithi079.stdout:1251: read oid 16 snap -1 2024-04-24T02:00:15.121 INFO:tasks.rados.rados.0.smithi079.stdout:1251: expect deleted 2024-04-24T02:00:15.122 INFO:tasks.rados.rados.0.smithi079.stdout:1252: snap_remove snap 140 2024-04-24T02:00:15.639 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T02:00:15.746 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:15.743+0000 7f18ebb9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:15.749 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:15.748+0000 7f18ebb9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:16.113 INFO:tasks.rados.rados.0.smithi079.stdout:1252: done (11 left) 2024-04-24T02:00:16.113 INFO:tasks.rados.rados.0.smithi079.stdout:1253: read oid 6 snap -1 2024-04-24T02:00:16.114 INFO:tasks.rados.rados.0.smithi079.stdout:1253: expect (ObjNum 567 snap 130 seq_num 567) 2024-04-24T02:00:16.114 INFO:tasks.rados.rados.0.smithi079.stdout:1254: snap_remove snap 137 2024-04-24T02:00:17.125 INFO:tasks.rados.rados.0.smithi079.stdout:1254: done (12 left) 2024-04-24T02:00:17.125 INFO:tasks.rados.rados.0.smithi079.stdout:1255: snap_remove snap 131 2024-04-24T02:00:17.847 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T02:00:17.965 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:17.963+0000 7f3557ced640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:17.969 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:17.969+0000 7f3557ced640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:18.130 INFO:tasks.rados.rados.0.smithi079.stdout:1255: done (12 left) 2024-04-24T02:00:18.130 INFO:tasks.rados.rados.0.smithi079.stdout:1256: read oid 17 snap -1 2024-04-24T02:00:18.130 INFO:tasks.rados.rados.0.smithi079.stdout:1256: expect (ObjNum 587 snap 137 seq_num 587) 2024-04-24T02:00:18.131 INFO:tasks.rados.rados.0.smithi079.stdout:1257: read oid 39 snap -1 2024-04-24T02:00:18.131 INFO:tasks.rados.rados.0.smithi079.stdout:1257: expect (ObjNum 580 snap 135 seq_num 580) 2024-04-24T02:00:18.131 INFO:tasks.rados.rados.0.smithi079.stdout:1258: write oid 10 current snap is 142 2024-04-24T02:00:18.131 INFO:tasks.rados.rados.0.smithi079.stdout:1243: finishing write tid 2 to smithi07929762-28 2024-04-24T02:00:18.131 INFO:tasks.rados.rados.0.smithi079.stdout:1243: finishing write tid 3 to smithi07929762-28 2024-04-24T02:00:18.131 INFO:tasks.rados.rados.0.smithi079.stdout:1243: finishing write tid 4 to smithi07929762-28 2024-04-24T02:00:18.131 INFO:tasks.rados.rados.0.smithi079.stdout:1243: oid 28 updating version 0 to 335 2024-04-24T02:00:18.131 INFO:tasks.rados.rados.0.smithi079.stdout:1243: oid 28 version 335 is already newer than 335 2024-04-24T02:00:18.131 INFO:tasks.rados.rados.0.smithi079.stdout:1243: oid 28 version 335 is already newer than 333 2024-04-24T02:00:18.131 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 28 v 335 (ObjNum 607 snap 142 seq_num 607) dirty exists 2024-04-24T02:00:18.131 INFO:tasks.rados.rados.0.smithi079.stdout:1243: left oid 28 (ObjNum 607 snap 142 seq_num 607) 2024-04-24T02:00:18.131 INFO:tasks.rados.rados.0.smithi079.stdout:1240: expect (ObjNum 603 snap 141 seq_num 603) 2024-04-24T02:00:18.273 INFO:tasks.rados.rados.0.smithi079.stdout:1245: finishing write tid 1 to smithi07929762-40 2024-04-24T02:00:18.273 INFO:tasks.rados.rados.0.smithi079.stdout:1241: expect (ObjNum 499 snap 112 seq_num 499) 2024-04-24T02:00:18.376 INFO:tasks.rados.rados.0.smithi079.stdout:1245: finishing write tid 2 to smithi07929762-40 2024-04-24T02:00:18.376 INFO:tasks.rados.rados.0.smithi079.stdout:1245: finishing write tid 3 to smithi07929762-40 2024-04-24T02:00:18.376 INFO:tasks.rados.rados.0.smithi079.stdout:1258: seq_num 611 ranges {669077=485379,1587792=681285} 2024-04-24T02:00:18.383 INFO:tasks.rados.rados.0.smithi079.stdout:1258: writing smithi07929762-10 from 669077 to 1154456 tid 1 2024-04-24T02:00:18.388 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-04-24T02:00:18.388 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-04-24T02:00:18.388 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-04-24T02:00:18.389 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-04-24T02:00:18.390 INFO:tasks.rados.rados.0.smithi079.stdout:1258: writing smithi07929762-10 from 1587792 to 2269077 tid 2 2024-04-24T02:00:18.391 INFO:tasks.rados.rados.0.smithi079.stdout:1240: done (14 left) 2024-04-24T02:00:18.392 INFO:tasks.rados.rados.0.smithi079.stdout:1241: done (13 left) 2024-04-24T02:00:18.392 INFO:tasks.rados.rados.0.smithi079.stdout:1242: done (12 left) 2024-04-24T02:00:18.392 INFO:tasks.rados.rados.0.smithi079.stdout:1243: done (11 left) 2024-04-24T02:00:18.392 INFO:tasks.rados.rados.0.smithi079.stdout:1244: done (10 left) 2024-04-24T02:00:18.392 INFO:tasks.rados.rados.0.smithi079.stdout:1259: read oid 4 snap 136 2024-04-24T02:00:18.392 INFO:tasks.rados.rados.0.smithi079.stdout:1259: expect (ObjNum 471 snap 107 seq_num 471) 2024-04-24T02:00:18.392 INFO:tasks.rados.rados.0.smithi079.stdout:1260: read oid 50 snap -1 2024-04-24T02:00:18.392 INFO:tasks.rados.rados.0.smithi079.stdout:1260: expect deleted 2024-04-24T02:00:18.392 INFO:tasks.rados.rados.0.smithi079.stdout:1261: delete oid 12 current snap is 142 2024-04-24T02:00:18.392 INFO:tasks.rados.rados.0.smithi079.stdout:1245: finishing write tid 4 to smithi07929762-40 2024-04-24T02:00:18.392 INFO:tasks.rados.rados.0.smithi079.stdout:1245: oid 40 updating version 0 to 383 2024-04-24T02:00:18.392 INFO:tasks.rados.rados.0.smithi079.stdout:1245: oid 40 version 383 is already newer than 383 2024-04-24T02:00:18.392 INFO:tasks.rados.rados.0.smithi079.stdout:1245: oid 40 version 383 is already newer than 383 2024-04-24T02:00:18.392 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 40 v 383 (ObjNum 608 snap 142 seq_num 608) dirty exists 2024-04-24T02:00:18.393 INFO:tasks.rados.rados.0.smithi079.stdout:1245: left oid 40 (ObjNum 608 snap 142 seq_num 608) 2024-04-24T02:00:18.393 INFO:tasks.rados.rados.0.smithi079.stdout:1246: finishing write tid 1 to smithi07929762-48 2024-04-24T02:00:18.393 INFO:tasks.rados.rados.0.smithi079.stdout:1246: finishing write tid 2 to smithi07929762-48 2024-04-24T02:00:18.393 INFO:tasks.rados.rados.0.smithi079.stdout:1246: finishing write tid 3 to smithi07929762-48 2024-04-24T02:00:18.393 INFO:tasks.rados.rados.0.smithi079.stdout:1246: finishing write tid 4 to smithi07929762-48 2024-04-24T02:00:18.393 INFO:tasks.rados.rados.0.smithi079.stdout:1246: oid 48 updating version 0 to 224 2024-04-24T02:00:18.393 INFO:tasks.rados.rados.0.smithi079.stdout:1246: oid 48 version 224 is already newer than 222 2024-04-24T02:00:18.393 INFO:tasks.rados.rados.0.smithi079.stdout:1246: oid 48 version 224 is already newer than 224 2024-04-24T02:00:18.393 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 48 v 224 (ObjNum 609 snap 142 seq_num 609) dirty exists 2024-04-24T02:00:18.393 INFO:tasks.rados.rados.0.smithi079.stdout:1246: left oid 48 (ObjNum 609 snap 142 seq_num 609) 2024-04-24T02:00:18.393 INFO:tasks.rados.rados.0.smithi079.stdout:1249: finishing write tid 1 to smithi07929762-27 2024-04-24T02:00:18.393 INFO:tasks.rados.rados.0.smithi079.stdout:1250: expect (ObjNum 500 snap 112 seq_num 500) 2024-04-24T02:00:18.501 INFO:tasks.rados.rados.0.smithi079.stdout:1249: finishing write tid 2 to smithi07929762-27 2024-04-24T02:00:18.501 INFO:tasks.rados.rados.0.smithi079.stdout:1249: finishing write tid 3 to smithi07929762-27 2024-04-24T02:00:18.502 INFO:tasks.rados.rados.0.smithi079.stdout:1249: finishing write tid 4 to smithi07929762-27 2024-04-24T02:00:18.502 INFO:tasks.rados.rados.0.smithi079.stdout:1249: oid 27 updating version 0 to 112 2024-04-24T02:00:18.502 INFO:tasks.rados.rados.0.smithi079.stdout:1249: oid 27 version 112 is already newer than 112 2024-04-24T02:00:18.502 INFO:tasks.rados.rados.0.smithi079.stdout:1249: oid 27 version 112 is already newer than 112 2024-04-24T02:00:18.502 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 27 v 112 (ObjNum 610 snap 142 seq_num 610) dirty exists 2024-04-24T02:00:18.502 INFO:tasks.rados.rados.0.smithi079.stdout:1249: left oid 27 (ObjNum 610 snap 142 seq_num 610) 2024-04-24T02:00:18.502 INFO:tasks.rados.rados.0.smithi079.stdout:1245: done (12 left) 2024-04-24T02:00:18.502 INFO:tasks.rados.rados.0.smithi079.stdout:1246: done (11 left) 2024-04-24T02:00:18.502 INFO:tasks.rados.rados.0.smithi079.stdout:1248: done (10 left) 2024-04-24T02:00:18.502 INFO:tasks.rados.rados.0.smithi079.stdout:1249: done (9 left) 2024-04-24T02:00:18.502 INFO:tasks.rados.rados.0.smithi079.stdout:1250: done (8 left) 2024-04-24T02:00:18.502 INFO:tasks.rados.rados.0.smithi079.stdout:1251: done (7 left) 2024-04-24T02:00:18.502 INFO:tasks.rados.rados.0.smithi079.stdout:1261: done (6 left) 2024-04-24T02:00:18.502 INFO:tasks.rados.rados.0.smithi079.stdout:1262: read oid 35 snap -1 2024-04-24T02:00:18.503 INFO:tasks.rados.rados.0.smithi079.stdout:1262: expect (ObjNum 572 snap 131 seq_num 572) 2024-04-24T02:00:18.503 INFO:tasks.rados.rados.0.smithi079.stdout:1263: snap_create 2024-04-24T02:00:18.503 INFO:tasks.rados.rados.0.smithi079.stdout:1253: expect (ObjNum 567 snap 130 seq_num 567) 2024-04-24T02:00:18.530 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:18.524+0000 7f3f7e020640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:18.534 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:18.534+0000 7f3f7e020640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:18.549 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T02:00:18.549 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T02:00:18.549 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T02:00:18.549 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T02:00:18.549 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T02:00:18.549 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T02:00:18.549 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T02:00:18.549 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T02:00:18.550 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T02:00:18.550 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T02:00:18.550 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T02:00:18.550 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T02:00:18.553 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T02:00:18.553 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T02:00:18.554 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-04-24T02:00:18.663 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:18.657+0000 7f085819d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:18.667 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:18.666+0000 7f085819d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:18.681 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T02:00:18.682 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T02:00:18.682 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T02:00:18.682 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T02:00:18.682 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T02:00:18.682 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T02:00:18.683 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T02:00:18.683 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T02:00:18.683 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T02:00:18.683 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T02:00:18.683 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T02:00:18.683 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T02:00:18.685 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T02:00:18.686 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T02:00:18.686 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-04-24T02:00:18.796 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:18.790+0000 7f0dced9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:18.800 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:18.799+0000 7f0dced9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:18.806 INFO:tasks.rados.rados.0.smithi079.stdout:1256: expect (ObjNum 587 snap 137 seq_num 587) 2024-04-24T02:00:18.816 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T02:00:18.816 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T02:00:18.816 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T02:00:18.816 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T02:00:18.816 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T02:00:18.816 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T02:00:18.817 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T02:00:18.817 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T02:00:18.817 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T02:00:18.817 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T02:00:18.817 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T02:00:18.817 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T02:00:18.819 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T02:00:18.819 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T02:00:18.820 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-04-24T02:00:18.928 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:18.922+0000 7efe73b1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:18.932 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:18.931+0000 7efe73b1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:18.948 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T02:00:18.948 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T02:00:18.949 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T02:00:18.949 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T02:00:18.949 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T02:00:18.949 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T02:00:18.949 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T02:00:18.949 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T02:00:18.949 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T02:00:18.949 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T02:00:18.949 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T02:00:18.950 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T02:00:18.951 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T02:00:18.951 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T02:00:18.960 INFO:tasks.rados.rados.0.smithi079.stdout:1257: expect (ObjNum 580 snap 135 seq_num 580) 2024-04-24T02:00:19.101 INFO:tasks.rados.rados.0.smithi079.stdout:1259: expect (ObjNum 471 snap 107 seq_num 471) 2024-04-24T02:00:19.303 INFO:tasks.rados.rados.0.smithi079.stdout:1258: finishing write tid 1 to smithi07929762-10 2024-04-24T02:00:19.303 INFO:tasks.rados.rados.0.smithi079.stdout:1258: finishing write tid 2 to smithi07929762-10 2024-04-24T02:00:19.304 INFO:tasks.rados.rados.0.smithi079.stdout:1258: finishing write tid 3 to smithi07929762-10 2024-04-24T02:00:19.304 INFO:tasks.rados.rados.0.smithi079.stdout:1258: finishing write tid 4 to smithi07929762-10 2024-04-24T02:00:19.304 INFO:tasks.rados.rados.0.smithi079.stdout:1258: oid 10 updating version 0 to 346 2024-04-24T02:00:19.304 INFO:tasks.rados.rados.0.smithi079.stdout:1258: oid 10 version 346 is already newer than 346 2024-04-24T02:00:19.304 INFO:tasks.rados.rados.0.smithi079.stdout:1258: oid 10 version 346 is already newer than 346 2024-04-24T02:00:19.304 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 10 v 346 (ObjNum 611 snap 142 seq_num 611) dirty exists 2024-04-24T02:00:19.304 INFO:tasks.rados.rados.0.smithi079.stdout:1258: left oid 10 (ObjNum 611 snap 142 seq_num 611) 2024-04-24T02:00:19.304 INFO:tasks.rados.rados.0.smithi079.stdout:1253: done (7 left) 2024-04-24T02:00:19.304 INFO:tasks.rados.rados.0.smithi079.stdout:1256: done (6 left) 2024-04-24T02:00:19.304 INFO:tasks.rados.rados.0.smithi079.stdout:1257: done (5 left) 2024-04-24T02:00:19.304 INFO:tasks.rados.rados.0.smithi079.stdout:1258: done (4 left) 2024-04-24T02:00:19.304 INFO:tasks.rados.rados.0.smithi079.stdout:1259: done (3 left) 2024-04-24T02:00:19.304 INFO:tasks.rados.rados.0.smithi079.stdout:1260: done (2 left) 2024-04-24T02:00:19.304 INFO:tasks.rados.rados.0.smithi079.stdout:1263: done (1 left) 2024-04-24T02:00:19.304 INFO:tasks.rados.rados.0.smithi079.stdout:1264: write (excl) oid 19 current snap is 143 2024-04-24T02:00:19.305 INFO:tasks.rados.rados.0.smithi079.stdout:1262: expect (ObjNum 572 snap 131 seq_num 572) 2024-04-24T02:00:19.578 INFO:tasks.rados.rados.0.smithi079.stdout:1264: seq_num 613 ranges {702691=635877,2104756=197935} 2024-04-24T02:00:19.584 INFO:tasks.rados.rados.0.smithi079.stdout:1264: writing smithi07929762-19 from 702691 to 1338568 tid 1 2024-04-24T02:00:19.590 INFO:tasks.rados.rados.0.smithi079.stdout:1264: writing smithi07929762-19 from 2104756 to 2302691 tid 2 2024-04-24T02:00:19.590 INFO:tasks.rados.rados.0.smithi079.stdout:1262: done (1 left) 2024-04-24T02:00:19.590 INFO:tasks.rados.rados.0.smithi079.stdout:1265: write (excl) oid 40 current snap is 143 2024-04-24T02:00:19.590 INFO:tasks.rados.rados.0.smithi079.stdout:1265: seq_num 614 ranges {719498=711126,2046885=272613} 2024-04-24T02:00:19.599 INFO:tasks.rados.rados.0.smithi079.stdout:1265: writing smithi07929762-40 from 719498 to 1430624 tid 1 2024-04-24T02:00:19.604 INFO:tasks.rados.rados.0.smithi079.stdout:1265: writing smithi07929762-40 from 2046885 to 2319498 tid 2 2024-04-24T02:00:19.604 INFO:tasks.rados.rados.0.smithi079.stdout:1266: write oid 31 current snap is 143 2024-04-24T02:00:19.605 INFO:tasks.rados.rados.0.smithi079.stdout:1266: seq_num 615 ranges {736305=786375,2105367=230938} 2024-04-24T02:00:19.613 INFO:tasks.rados.rados.0.smithi079.stdout:1266: writing smithi07929762-31 from 736305 to 1522680 tid 1 2024-04-24T02:00:19.618 INFO:tasks.rados.rados.0.smithi079.stdout:1266: writing smithi07929762-31 from 2105367 to 2336305 tid 2 2024-04-24T02:00:19.618 INFO:tasks.rados.rados.0.smithi079.stdout:1267: read oid 44 snap -1 2024-04-24T02:00:19.618 INFO:tasks.rados.rados.0.smithi079.stdout:1267: expect deleted 2024-04-24T02:00:19.618 INFO:tasks.rados.rados.0.smithi079.stdout:1268: write oid 41 current snap is 143 2024-04-24T02:00:19.618 INFO:tasks.rados.rados.0.smithi079.stdout:1268: seq_num 616 ranges {753112=577977,1880202=472910} 2024-04-24T02:00:19.627 INFO:tasks.rados.rados.0.smithi079.stdout:1268: writing smithi07929762-41 from 753112 to 1331089 tid 1 2024-04-24T02:00:19.633 INFO:tasks.rados.rados.0.smithi079.stdout:1268: writing smithi07929762-41 from 1880202 to 2353112 tid 2 2024-04-24T02:00:19.634 INFO:tasks.rados.rados.0.smithi079.stdout:1264: finishing write tid 1 to smithi07929762-19 2024-04-24T02:00:19.634 INFO:tasks.rados.rados.0.smithi079.stdout:1269: write oid 3 current snap is 143 2024-04-24T02:00:19.634 INFO:tasks.rados.rados.0.smithi079.stdout:1269: seq_num 617 ranges {769919=653226,1938684=431235} 2024-04-24T02:00:19.643 INFO:tasks.rados.rados.0.smithi079.stdout:1269: writing smithi07929762-3 from 769919 to 1423145 tid 1 2024-04-24T02:00:19.649 INFO:tasks.rados.rados.0.smithi079.stdout:1269: writing smithi07929762-3 from 1938684 to 2369919 tid 2 2024-04-24T02:00:19.649 INFO:tasks.rados.rados.0.smithi079.stdout:1267: done (5 left) 2024-04-24T02:00:19.649 INFO:tasks.rados.rados.0.smithi079.stdout:1270: write (excl) oid 44 current snap is 143 2024-04-24T02:00:19.649 INFO:tasks.rados.rados.0.smithi079.stdout:1270: seq_num 618 ranges {786726=728475,2280813=105913} 2024-04-24T02:00:19.658 INFO:tasks.rados.rados.0.smithi079.stdout:1270: writing smithi07929762-44 from 786726 to 1515201 tid 1 2024-04-24T02:00:19.664 INFO:tasks.rados.rados.0.smithi079.stdout:1270: writing smithi07929762-44 from 2280813 to 2386726 tid 2 2024-04-24T02:00:19.664 INFO:tasks.rados.rados.0.smithi079.stdout:1271: snap_remove snap 142 2024-04-24T02:00:20.066 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T02:00:20.138 INFO:tasks.rados.rados.0.smithi079.stdout:1271: done (6 left) 2024-04-24T02:00:20.139 INFO:tasks.rados.rados.0.smithi079.stdout:1272: write oid 13 current snap is 143 2024-04-24T02:00:20.139 INFO:tasks.rados.rados.0.smithi079.stdout:1266: finishing write tid 1 to smithi07929762-31 2024-04-24T02:00:20.139 INFO:tasks.rados.rados.0.smithi079.stdout:1272: seq_num 619 ranges {403533=403724,1539295=652201} 2024-04-24T02:00:20.143 INFO:tasks.rados.rados.0.smithi079.stdout:1272: writing smithi07929762-13 from 403533 to 807257 tid 1 2024-04-24T02:00:20.151 INFO:tasks.rados.rados.0.smithi079.stdout:1272: writing smithi07929762-13 from 1539295 to 2191496 tid 2 2024-04-24T02:00:20.151 INFO:tasks.rados.rados.0.smithi079.stdout:1273: write (excl) oid 30 current snap is 143 2024-04-24T02:00:20.151 INFO:tasks.rados.rados.0.smithi079.stdout:1273: seq_num 620 ranges {420340=478973,1597777=512212} 2024-04-24T02:00:20.155 INFO:tasks.rados.rados.0.smithi079.stdout:1273: writing smithi07929762-30 from 420340 to 899313 tid 1 2024-04-24T02:00:20.163 INFO:tasks.rados.rados.0.smithi079.stdout:1273: writing smithi07929762-30 from 1597777 to 2109989 tid 2 2024-04-24T02:00:20.163 INFO:tasks.rados.rados.0.smithi079.stdout:1274: read oid 14 snap -1 2024-04-24T02:00:20.163 INFO:tasks.rados.rados.0.smithi079.stdout:1274: expect (ObjNum 506 snap 112 seq_num 506) 2024-04-24T02:00:20.164 INFO:tasks.rados.rados.0.smithi079.stdout:1275: write oid 22 current snap is 143 2024-04-24T02:00:20.164 INFO:tasks.rados.rados.0.smithi079.stdout:1264: finishing write tid 2 to smithi07929762-19 2024-04-24T02:00:20.164 INFO:tasks.rados.rados.0.smithi079.stdout:1264: finishing write tid 3 to smithi07929762-19 2024-04-24T02:00:20.164 INFO:tasks.rados.rados.0.smithi079.stdout:1268: finishing write tid 1 to smithi07929762-41 2024-04-24T02:00:20.164 INFO:tasks.rados.rados.0.smithi079.stdout:1266: finishing write tid 2 to smithi07929762-31 2024-04-24T02:00:20.164 INFO:tasks.rados.rados.0.smithi079.stdout:1270: finishing write tid 1 to smithi07929762-44 2024-04-24T02:00:20.164 INFO:tasks.rados.rados.0.smithi079.stdout:1266: finishing write tid 3 to smithi07929762-31 2024-04-24T02:00:20.164 INFO:tasks.rados.rados.0.smithi079.stdout:1264: finishing write tid 4 to smithi07929762-19 2024-04-24T02:00:20.164 INFO:tasks.rados.rados.0.smithi079.stdout:1264: oid 19 updating version 0 to 338 2024-04-24T02:00:20.164 INFO:tasks.rados.rados.0.smithi079.stdout:1264: oid 19 updating version 338 to 340 2024-04-24T02:00:20.164 INFO:tasks.rados.rados.0.smithi079.stdout:1264: oid 19 version 340 is already newer than 340 2024-04-24T02:00:20.164 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 19 v 340 (ObjNum 613 snap 143 seq_num 613) dirty exists 2024-04-24T02:00:20.164 INFO:tasks.rados.rados.0.smithi079.stdout:1264: left oid 19 (ObjNum 613 snap 143 seq_num 613) 2024-04-24T02:00:20.164 INFO:tasks.rados.rados.0.smithi079.stdout:1265: finishing write tid 1 to smithi07929762-40 2024-04-24T02:00:20.164 INFO:tasks.rados.rados.0.smithi079.stdout:1268: finishing write tid 2 to smithi07929762-41 2024-04-24T02:00:20.165 INFO:tasks.rados.rados.0.smithi079.stdout:1266: finishing write tid 4 to smithi07929762-31 2024-04-24T02:00:20.165 INFO:tasks.rados.rados.0.smithi079.stdout:1266: oid 31 updating version 0 to 582 2024-04-24T02:00:20.165 INFO:tasks.rados.rados.0.smithi079.stdout:1266: oid 31 version 582 is already newer than 582 2024-04-24T02:00:20.165 INFO:tasks.rados.rados.0.smithi079.stdout:1266: oid 31 version 582 is already newer than 582 2024-04-24T02:00:20.165 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 31 v 582 (ObjNum 615 snap 143 seq_num 615) dirty exists 2024-04-24T02:00:20.165 INFO:tasks.rados.rados.0.smithi079.stdout:1266: left oid 31 (ObjNum 615 snap 143 seq_num 615) 2024-04-24T02:00:20.165 INFO:tasks.rados.rados.0.smithi079.stdout:1268: finishing write tid 3 to smithi07929762-41 2024-04-24T02:00:20.165 INFO:tasks.rados.rados.0.smithi079.stdout:1268: finishing write tid 4 to smithi07929762-41 2024-04-24T02:00:20.165 INFO:tasks.rados.rados.0.smithi079.stdout:1268: oid 41 updating version 0 to 177 2024-04-24T02:00:20.165 INFO:tasks.rados.rados.0.smithi079.stdout:1268: oid 41 version 177 is already newer than 177 2024-04-24T02:00:20.165 INFO:tasks.rados.rados.0.smithi079.stdout:1268: oid 41 version 177 is already newer than 177 2024-04-24T02:00:20.165 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 41 v 177 (ObjNum 616 snap 143 seq_num 616) dirty exists 2024-04-24T02:00:20.165 INFO:tasks.rados.rados.0.smithi079.stdout:1268: left oid 41 (ObjNum 616 snap 143 seq_num 616) 2024-04-24T02:00:20.165 INFO:tasks.rados.rados.0.smithi079.stdout:1270: finishing write tid 2 to smithi07929762-44 2024-04-24T02:00:20.165 INFO:tasks.rados.rados.0.smithi079.stdout:1270: finishing write tid 3 to smithi07929762-44 2024-04-24T02:00:20.166 INFO:tasks.rados.rados.0.smithi079.stdout:1275: seq_num 621 ranges {437147=554222,1656259=655870} 2024-04-24T02:00:20.168 INFO:tasks.rados.rados.0.smithi079.stdout:1275: writing smithi07929762-22 from 437147 to 991369 tid 1 2024-04-24T02:00:20.176 INFO:tasks.rados.rados.0.smithi079.stdout:1275: writing smithi07929762-22 from 1656259 to 2312129 tid 2 2024-04-24T02:00:20.176 INFO:tasks.rados.rados.0.smithi079.stdout:1264: done (9 left) 2024-04-24T02:00:20.176 INFO:tasks.rados.rados.0.smithi079.stdout:1266: done (8 left) 2024-04-24T02:00:20.176 INFO:tasks.rados.rados.0.smithi079.stdout:1268: done (7 left) 2024-04-24T02:00:20.176 INFO:tasks.rados.rados.0.smithi079.stdout:1276: snap_remove snap 138 2024-04-24T02:00:20.185 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:20.182+0000 7f0ca259d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:20.189 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:20.189+0000 7f0ca259d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:21.142 INFO:tasks.rados.rados.0.smithi079.stdout:1276: done (7 left) 2024-04-24T02:00:21.143 INFO:tasks.rados.rados.0.smithi079.stdout:1277: delete oid 6 current snap is 143 2024-04-24T02:00:21.143 INFO:tasks.rados.rados.0.smithi079.stdout:1265: finishing write tid 2 to smithi07929762-40 2024-04-24T02:00:21.143 INFO:tasks.rados.rados.0.smithi079.stdout:1270: finishing write tid 4 to smithi07929762-44 2024-04-24T02:00:21.143 INFO:tasks.rados.rados.0.smithi079.stdout:1270: oid 44 updating version 0 to 275 2024-04-24T02:00:21.143 INFO:tasks.rados.rados.0.smithi079.stdout:1270: oid 44 version 275 is already newer than 275 2024-04-24T02:00:21.143 INFO:tasks.rados.rados.0.smithi079.stdout:1270: oid 44 version 275 is already newer than 273 2024-04-24T02:00:21.143 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 44 v 275 (ObjNum 618 snap 143 seq_num 618) dirty exists 2024-04-24T02:00:21.143 INFO:tasks.rados.rados.0.smithi079.stdout:1270: left oid 44 (ObjNum 618 snap 143 seq_num 618) 2024-04-24T02:00:21.143 INFO:tasks.rados.rados.0.smithi079.stdout:1269: finishing write tid 1 to smithi07929762-3 2024-04-24T02:00:21.143 INFO:tasks.rados.rados.0.smithi079.stdout:1265: finishing write tid 3 to smithi07929762-40 2024-04-24T02:00:21.143 INFO:tasks.rados.rados.0.smithi079.stdout:1265: finishing write tid 4 to smithi07929762-40 2024-04-24T02:00:21.143 INFO:tasks.rados.rados.0.smithi079.stdout:1265: oid 40 updating version 0 to 393 2024-04-24T02:00:21.143 INFO:tasks.rados.rados.0.smithi079.stdout:1265: oid 40 version 393 is already newer than 391 2024-04-24T02:00:21.143 INFO:tasks.rados.rados.0.smithi079.stdout:1265: oid 40 version 393 is already newer than 393 2024-04-24T02:00:21.143 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 40 v 393 (ObjNum 614 snap 143 seq_num 614) dirty exists 2024-04-24T02:00:21.144 INFO:tasks.rados.rados.0.smithi079.stdout:1265: left oid 40 (ObjNum 614 snap 143 seq_num 614) 2024-04-24T02:00:21.144 INFO:tasks.rados.rados.0.smithi079.stdout:1269: finishing write tid 2 to smithi07929762-3 2024-04-24T02:00:21.144 INFO:tasks.rados.rados.0.smithi079.stdout:1269: finishing write tid 3 to smithi07929762-3 2024-04-24T02:00:21.144 INFO:tasks.rados.rados.0.smithi079.stdout:1269: finishing write tid 4 to smithi07929762-3 2024-04-24T02:00:21.144 INFO:tasks.rados.rados.0.smithi079.stdout:1269: oid 3 updating version 0 to 585 2024-04-24T02:00:21.144 INFO:tasks.rados.rados.0.smithi079.stdout:1269: oid 3 version 585 is already newer than 583 2024-04-24T02:00:21.144 INFO:tasks.rados.rados.0.smithi079.stdout:1269: oid 3 version 585 is already newer than 585 2024-04-24T02:00:21.144 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 3 v 585 (ObjNum 617 snap 143 seq_num 617) dirty exists 2024-04-24T02:00:21.144 INFO:tasks.rados.rados.0.smithi079.stdout:1269: left oid 3 (ObjNum 617 snap 143 seq_num 617) 2024-04-24T02:00:21.144 INFO:tasks.rados.rados.0.smithi079.stdout:1274: expect (ObjNum 506 snap 112 seq_num 506) 2024-04-24T02:00:21.247 INFO:tasks.rados.rados.0.smithi079.stdout:1272: finishing write tid 1 to smithi07929762-13 2024-04-24T02:00:21.247 INFO:tasks.rados.rados.0.smithi079.stdout:1273: finishing write tid 1 to smithi07929762-30 2024-04-24T02:00:21.248 INFO:tasks.rados.rados.0.smithi079.stdout:1275: finishing write tid 1 to smithi07929762-22 2024-04-24T02:00:21.248 INFO:tasks.rados.rados.0.smithi079.stdout:1272: finishing write tid 2 to smithi07929762-13 2024-04-24T02:00:21.248 INFO:tasks.rados.rados.0.smithi079.stdout:1275: finishing write tid 2 to smithi07929762-22 2024-04-24T02:00:21.248 INFO:tasks.rados.rados.0.smithi079.stdout:1272: finishing write tid 3 to smithi07929762-13 2024-04-24T02:00:21.248 INFO:tasks.rados.rados.0.smithi079.stdout:1275: finishing write tid 3 to smithi07929762-22 2024-04-24T02:00:21.248 INFO:tasks.rados.rados.0.smithi079.stdout:1275: finishing write tid 4 to smithi07929762-22 2024-04-24T02:00:21.248 INFO:tasks.rados.rados.0.smithi079.stdout:1275: oid 22 updating version 0 to 180 2024-04-24T02:00:21.248 INFO:tasks.rados.rados.0.smithi079.stdout:1275: oid 22 version 180 is already newer than 180 2024-04-24T02:00:21.248 INFO:tasks.rados.rados.0.smithi079.stdout:1275: oid 22 version 180 is already newer than 180 2024-04-24T02:00:21.248 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 22 v 180 (ObjNum 621 snap 143 seq_num 621) dirty exists 2024-04-24T02:00:21.248 INFO:tasks.rados.rados.0.smithi079.stdout:1275: left oid 22 (ObjNum 621 snap 143 seq_num 621) 2024-04-24T02:00:21.248 INFO:tasks.rados.rados.0.smithi079.stdout:1272: finishing write tid 4 to smithi07929762-13 2024-04-24T02:00:21.248 INFO:tasks.rados.rados.0.smithi079.stdout:1272: oid 13 updating version 0 to 435 2024-04-24T02:00:21.248 INFO:tasks.rados.rados.0.smithi079.stdout:1272: oid 13 version 435 is already newer than 435 2024-04-24T02:00:21.249 INFO:tasks.rados.rados.0.smithi079.stdout:1272: oid 13 version 435 is already newer than 435 2024-04-24T02:00:21.249 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 13 v 435 (ObjNum 619 snap 143 seq_num 619) dirty exists 2024-04-24T02:00:21.249 INFO:tasks.rados.rados.0.smithi079.stdout:1272: left oid 13 (ObjNum 619 snap 143 seq_num 619) 2024-04-24T02:00:21.249 INFO:tasks.rados.rados.0.smithi079.stdout:1273: finishing write tid 2 to smithi07929762-30 2024-04-24T02:00:21.249 INFO:tasks.rados.rados.0.smithi079.stdout:1273: finishing write tid 3 to smithi07929762-30 2024-04-24T02:00:21.249 INFO:tasks.rados.rados.0.smithi079.stdout:1273: finishing write tid 4 to smithi07929762-30 2024-04-24T02:00:21.249 INFO:tasks.rados.rados.0.smithi079.stdout:1273: oid 30 updating version 0 to 153 2024-04-24T02:00:21.249 INFO:tasks.rados.rados.0.smithi079.stdout:1273: oid 30 version 153 is already newer than 153 2024-04-24T02:00:21.249 INFO:tasks.rados.rados.0.smithi079.stdout:1273: oid 30 version 153 is already newer than 151 2024-04-24T02:00:21.249 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 30 v 153 (ObjNum 620 snap 143 seq_num 620) dirty exists 2024-04-24T02:00:21.249 INFO:tasks.rados.rados.0.smithi079.stdout:1273: left oid 30 (ObjNum 620 snap 143 seq_num 620) 2024-04-24T02:00:21.264 INFO:tasks.rados.rados.0.smithi079.stdout:1265: done (7 left) 2024-04-24T02:00:21.264 INFO:tasks.rados.rados.0.smithi079.stdout:1269: done (6 left) 2024-04-24T02:00:21.264 INFO:tasks.rados.rados.0.smithi079.stdout:1270: done (5 left) 2024-04-24T02:00:21.264 INFO:tasks.rados.rados.0.smithi079.stdout:1272: done (4 left) 2024-04-24T02:00:21.264 INFO:tasks.rados.rados.0.smithi079.stdout:1273: done (3 left) 2024-04-24T02:00:21.264 INFO:tasks.rados.rados.0.smithi079.stdout:1274: done (2 left) 2024-04-24T02:00:21.264 INFO:tasks.rados.rados.0.smithi079.stdout:1275: done (1 left) 2024-04-24T02:00:21.264 INFO:tasks.rados.rados.0.smithi079.stdout:1277: done (0 left) 2024-04-24T02:00:21.264 INFO:tasks.rados.rados.0.smithi079.stdout:1278: snap_create 2024-04-24T02:00:22.149 INFO:tasks.rados.rados.0.smithi079.stdout:1278: done (0 left) 2024-04-24T02:00:22.149 INFO:tasks.rados.rados.0.smithi079.stdout:1279: snap_remove snap 136 2024-04-24T02:00:22.364 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T02:00:22.470 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:22.468+0000 7fb4a655d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:22.474 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:22.473+0000 7fb4a655d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:23.155 INFO:tasks.rados.rados.0.smithi079.stdout:1279: done (0 left) 2024-04-24T02:00:23.155 INFO:tasks.rados.rados.0.smithi079.stdout:1280: write (excl) oid 43 current snap is 144 2024-04-24T02:00:23.155 INFO:tasks.rados.rados.0.smithi079.stdout:1280: seq_num 623 ranges {470761=704720,1656870=659539} 2024-04-24T02:00:23.161 INFO:tasks.rados.rados.0.smithi079.stdout:1280: writing smithi07929762-43 from 470761 to 1175481 tid 1 2024-04-24T02:00:23.168 INFO:tasks.rados.rados.0.smithi079.stdout:1280: writing smithi07929762-43 from 1656870 to 2316409 tid 2 2024-04-24T02:00:23.168 INFO:tasks.rados.rados.0.smithi079.stdout:1281: snap_remove snap 143 2024-04-24T02:00:23.952 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-04-24T02:00:23.952 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-04-24T02:00:23.953 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T02:00:24.066 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:24.061+0000 7f688f75d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:24.070 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:24.069+0000 7f688f75d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:24.155 INFO:tasks.rados.rados.0.smithi079.stdout:1281: done (1 left) 2024-04-24T02:00:24.155 INFO:tasks.rados.rados.0.smithi079.stdout:1282: delete oid 28 current snap is 144 2024-04-24T02:00:24.155 INFO:tasks.rados.rados.0.smithi079.stdout:1280: finishing write tid 1 to smithi07929762-43 2024-04-24T02:00:24.155 INFO:tasks.rados.rados.0.smithi079.stdout:1280: finishing write tid 2 to smithi07929762-43 2024-04-24T02:00:24.155 INFO:tasks.rados.rados.0.smithi079.stdout:1280: finishing write tid 3 to smithi07929762-43 2024-04-24T02:00:24.155 INFO:tasks.rados.rados.0.smithi079.stdout:1280: finishing write tid 4 to smithi07929762-43 2024-04-24T02:00:24.156 INFO:tasks.rados.rados.0.smithi079.stdout:1280: oid 43 updating version 0 to 161 2024-04-24T02:00:24.156 INFO:tasks.rados.rados.0.smithi079.stdout:1280: oid 43 version 161 is already newer than 159 2024-04-24T02:00:24.156 INFO:tasks.rados.rados.0.smithi079.stdout:1280: oid 43 version 161 is already newer than 161 2024-04-24T02:00:24.156 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 43 v 161 (ObjNum 623 snap 144 seq_num 623) dirty exists 2024-04-24T02:00:24.156 INFO:tasks.rados.rados.0.smithi079.stdout:1280: left oid 43 (ObjNum 623 snap 144 seq_num 623) 2024-04-24T02:00:24.319 INFO:tasks.rados.rados.0.smithi079.stdout:1280: done (1 left) 2024-04-24T02:00:24.319 INFO:tasks.rados.rados.0.smithi079.stdout:1282: done (0 left) 2024-04-24T02:00:24.319 INFO:tasks.rados.rados.0.smithi079.stdout:1283: delete oid 45 current snap is 144 2024-04-24T02:00:24.370 INFO:tasks.rados.rados.0.smithi079.stdout:1283: done (0 left) 2024-04-24T02:00:24.371 INFO:tasks.rados.rados.0.smithi079.stdout:1284: read oid 27 snap -1 2024-04-24T02:00:24.371 INFO:tasks.rados.rados.0.smithi079.stdout:1284: expect (ObjNum 610 snap 142 seq_num 610) 2024-04-24T02:00:24.371 INFO:tasks.rados.rados.0.smithi079.stdout:1285: write (excl) oid 30 current snap is 144 2024-04-24T02:00:24.371 INFO:tasks.rados.rados.0.smithi079.stdout:1285: seq_num 626 ranges {521182=646820,1948669=523219} 2024-04-24T02:00:24.376 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T02:00:24.377 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-24T02:00:24.378 INFO:tasks.rados.rados.0.smithi079.stdout:1285: writing smithi07929762-30 from 521182 to 1168002 tid 1 2024-04-24T02:00:24.384 INFO:tasks.rados.rados.0.smithi079.stdout:1285: writing smithi07929762-30 from 1948669 to 2471888 tid 2 2024-04-24T02:00:24.384 INFO:tasks.rados.rados.0.smithi079.stdout:1286: delete oid 18 current snap is 144 2024-04-24T02:00:24.391 INFO:tasks.rados.rados.0.smithi079.stdout:1284: expect (ObjNum 610 snap 142 seq_num 610) 2024-04-24T02:00:24.397 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":453,"stamp":"2024-04-24T02:00:24.152138+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":95216913,"num_objects":59,"num_object_clones":19,"num_object_copies":177,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":2365,"num_read_kb":1654498,"num_write":2189,"num_write_kb":560234,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4218,"ondisk_log_size":4218,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":406248,"kb_used_data":264800,"kb_used_omap":0,"kb_used_meta":140544,"kb_avail":377081112,"statfs":{"total":386547056640,"available":386131058688,"internally_reserved":0,"allocated":271155200,"data_stored":281781019,"data_compressed":89889,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":143917056},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1344527,"num_objects":-3,"num_object_clones":-3,"num_object_copies":-9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":4480,"num_write":8,"num_write_kb":1994,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.015230"},"pg_stats":[{"pgid":"3.f","version":"301'234","reported_seq":1147,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-04-24T02:00:19.152168+0000","last_change":"2024-04-24T02:00:19.175598+0000","last_active":"2024-04-24T02:00:19.152168+0000","last_peered":"2024-04-24T02:00:19.152168+0000","last_clean":"2024-04-24T02:00:19.152168+0000","last_became_active":"2024-04-24T01:55:30.386726+0000","last_became_peered":"2024-04-24T01:55:30.386726+0000","last_unstale":"2024-04-24T02:00:19.152168+0000","last_undegraded":"2024-04-24T02:00:19.152168+0000","last_fullsized":"2024-04-24T02:00:19.152168+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":234,"log_dups_size":0,"ondisk_log_size":234,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":21,"snaptrim_duration":0,"stat_sum":{"num_bytes":4370084,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":172,"num_read_kb":152845,"num_write":111,"num_write_kb":32111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"301'346","reported_seq":1211,"reported_epoch":301,"state":"active+clean+snaptrim","last_fresh":"2024-04-24T02:00:18.416348+0000","last_change":"2024-04-24T02:00:18.162587+0000","last_active":"2024-04-24T02:00:18.416348+0000","last_peered":"2024-04-24T02:00:18.416348+0000","last_clean":"2024-04-24T02:00:18.416348+0000","last_became_active":"2024-04-24T01:55:30.321866+0000","last_became_peered":"2024-04-24T01:55:30.321866+0000","last_unstale":"2024-04-24T02:00:18.416348+0000","last_undegraded":"2024-04-24T02:00:18.416348+0000","last_fullsized":"2024-04-24T02:00:18.416348+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":346,"log_dups_size":0,"ondisk_log_size":346,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":30,"snaptrim_duration":0,"stat_sum":{"num_bytes":10723328,"num_objects":7,"num_object_clones":3,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":198,"num_read_kb":136088,"num_write":177,"num_write_kb":48071,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"264'229","reported_seq":1118,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-04-24T02:00:22.281176+0000","last_change":"2024-04-24T02:00:22.291141+0000","last_active":"2024-04-24T02:00:22.281176+0000","last_peered":"2024-04-24T02:00:22.281176+0000","last_clean":"2024-04-24T02:00:22.281176+0000","last_became_active":"2024-04-24T01:55:30.320186+0000","last_became_peered":"2024-04-24T01:55:30.320186+0000","last_unstale":"2024-04-24T02:00:22.281176+0000","last_undegraded":"2024-04-24T02:00:22.281176+0000","last_fullsized":"2024-04-24T02:00:22.281176+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":229,"log_dups_size":0,"ondisk_log_size":229,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":24,"snaptrim_duration":0,"stat_sum":{"num_bytes":5159914,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":137,"num_read_kb":94252,"num_write":124,"num_write_kb":34018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"301'288","reported_seq":1174,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-04-24T02:00:19.153781+0000","last_change":"2024-04-24T02:00:19.175212+0000","last_active":"2024-04-24T02:00:19.153781+0000","last_peered":"2024-04-24T02:00:19.153781+0000","last_clean":"2024-04-24T02:00:19.153781+0000","last_became_active":"2024-04-24T01:55:30.288081+0000","last_became_peered":"2024-04-24T01:55:30.288081+0000","last_unstale":"2024-04-24T02:00:19.153781+0000","last_undegraded":"2024-04-24T02:00:19.153781+0000","last_fullsized":"2024-04-24T02:00:19.153781+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":288,"log_dups_size":0,"ondisk_log_size":288,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":28,"snaptrim_duration":0,"stat_sum":{"num_bytes":3395113,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":150,"num_read_kb":109442,"num_write":140,"num_write_kb":37310,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"301'116","reported_seq":1008,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-24T02:00:18.371541+0000","last_change":"2024-04-24T02:00:18.428587+0000","last_active":"2024-04-24T02:00:18.371541+0000","last_peered":"2024-04-24T02:00:18.371541+0000","last_clean":"2024-04-24T02:00:18.371541+0000","last_became_active":"2024-04-24T01:55:30.333815+0000","last_became_peered":"2024-04-24T01:55:30.333815+0000","last_unstale":"2024-04-24T02:00:18.371541+0000","last_undegraded":"2024-04-24T02:00:18.371541+0000","last_fullsized":"2024-04-24T02:00:18.371541+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":12,"snaptrim_duration":0,"stat_sum":{"num_bytes":2252270,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":61,"num_read_kb":37734,"num_write":66,"num_write_kb":17808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"301'579","reported_seq":1410,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-04-24T02:00:19.158784+0000","last_change":"2024-04-24T02:00:19.175979+0000","last_active":"2024-04-24T02:00:19.158784+0000","last_peered":"2024-04-24T02:00:19.158784+0000","last_clean":"2024-04-24T02:00:19.158784+0000","last_became_active":"2024-04-24T01:55:30.315075+0000","last_became_peered":"2024-04-24T01:55:30.315075+0000","last_unstale":"2024-04-24T02:00:19.158784+0000","last_undegraded":"2024-04-24T02:00:19.158784+0000","last_fullsized":"2024-04-24T02:00:19.158784+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":579,"log_dups_size":0,"ondisk_log_size":579,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":56,"snaptrim_duration":0,"stat_sum":{"num_bytes":2386693,"num_objects":7,"num_object_clones":3,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":265,"num_read_kb":173935,"num_write":255,"num_write_kb":65993,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"301'113","reported_seq":1022,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-04-24T02:00:19.160848+0000","last_change":"2024-04-24T02:00:18.383817+0000","last_active":"2024-04-24T02:00:19.160848+0000","last_peered":"2024-04-24T02:00:19.160848+0000","last_clean":"2024-04-24T02:00:19.160848+0000","last_became_active":"2024-04-24T01:55:30.350300+0000","last_became_peered":"2024-04-24T01:55:30.350300+0000","last_unstale":"2024-04-24T02:00:19.160848+0000","last_undegraded":"2024-04-24T02:00:19.160848+0000","last_fullsized":"2024-04-24T02:00:19.160848+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":10,"snaptrim_duration":0,"stat_sum":{"num_bytes":7647350,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":61,"num_read_kb":40457,"num_write":73,"num_write_kb":18631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"264'34","reported_seq":928,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-04-24T02:00:22.298055+0000","last_change":"2024-04-24T02:00:22.308838+0000","last_active":"2024-04-24T02:00:22.298055+0000","last_peered":"2024-04-24T02:00:22.298055+0000","last_clean":"2024-04-24T02:00:22.298055+0000","last_became_active":"2024-04-24T01:55:30.376813+0000","last_became_peered":"2024-04-24T01:55:30.376813+0000","last_unstale":"2024-04-24T02:00:22.298055+0000","last_undegraded":"2024-04-24T02:00:22.298055+0000","last_fullsized":"2024-04-24T02:00:22.298055+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0,"stat_sum":{"num_bytes":3025359,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":10082,"num_write":27,"num_write_kb":6802,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'97","reported_seq":899,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-24T02:00:18.346264+0000","last_change":"2024-04-24T01:55:16.196470+0000","last_active":"2024-04-24T02:00:18.346264+0000","last_peered":"2024-04-24T02:00:18.346264+0000","last_clean":"2024-04-24T02:00:18.346264+0000","last_became_active":"2024-04-24T01:55:16.190464+0000","last_became_peered":"2024-04-24T01:55:16.190464+0000","last_unstale":"2024-04-24T02:00:18.346264+0000","last_undegraded":"2024-04-24T02:00:18.346264+0000","last_fullsized":"2024-04-24T02:00:18.346264+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:15.127115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:15.127115+0000","last_clean_scrub_stamp":"2024-04-24T01:55:15.127115+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":762,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-04-24T02:00:22.296393+0000","last_change":"2024-04-24T01:55:18.236148+0000","last_active":"2024-04-24T02:00:22.296393+0000","last_peered":"2024-04-24T02:00:22.296393+0000","last_clean":"2024-04-24T02:00:22.296393+0000","last_became_active":"2024-04-24T01:55:18.230154+0000","last_became_peered":"2024-04-24T01:55:18.230154+0000","last_unstale":"2024-04-24T02:00:22.296393+0000","last_undegraded":"2024-04-24T02:00:22.296393+0000","last_fullsized":"2024-04-24T02:00:22.296393+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"305'437","reported_seq":1315,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-04-24T02:00:22.314490+0000","last_change":"2024-04-24T02:00:22.353909+0000","last_active":"2024-04-24T02:00:22.314490+0000","last_peered":"2024-04-24T02:00:22.314490+0000","last_clean":"2024-04-24T02:00:22.314490+0000","last_became_active":"2024-04-24T01:55:30.316222+0000","last_became_peered":"2024-04-24T01:55:30.316222+0000","last_unstale":"2024-04-24T02:00:22.314490+0000","last_undegraded":"2024-04-24T02:00:22.314490+0000","last_fullsized":"2024-04-24T02:00:22.314490+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":437,"log_dups_size":0,"ondisk_log_size":437,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":37,"snaptrim_duration":0,"stat_sum":{"num_bytes":6202014,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":230,"num_read_kb":163232,"num_write":230,"num_write_kb":64433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":777,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-04-24T02:00:19.163756+0000","last_change":"2024-04-24T01:55:18.265905+0000","last_active":"2024-04-24T02:00:19.163756+0000","last_peered":"2024-04-24T02:00:19.163756+0000","last_clean":"2024-04-24T02:00:19.163756+0000","last_became_active":"2024-04-24T01:55:18.257020+0000","last_became_peered":"2024-04-24T01:55:18.257020+0000","last_unstale":"2024-04-24T02:00:19.163756+0000","last_undegraded":"2024-04-24T02:00:19.163756+0000","last_fullsized":"2024-04-24T02:00:19.163756+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"289'337","reported_seq":1191,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-24T02:00:18.528572+0000","last_change":"2024-04-24T02:00:18.358147+0000","last_active":"2024-04-24T02:00:18.528572+0000","last_peered":"2024-04-24T02:00:18.528572+0000","last_clean":"2024-04-24T02:00:18.528572+0000","last_became_active":"2024-04-24T01:55:30.344129+0000","last_became_peered":"2024-04-24T01:55:30.344129+0000","last_unstale":"2024-04-24T02:00:18.528572+0000","last_undegraded":"2024-04-24T02:00:18.528572+0000","last_fullsized":"2024-04-24T02:00:18.528572+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":337,"log_dups_size":0,"ondisk_log_size":337,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":29,"snaptrim_duration":0,"stat_sum":{"num_bytes":3613604,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":141,"num_read_kb":82335,"num_write":159,"num_write_kb":44990,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":762,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-04-24T02:00:22.277576+0000","last_change":"2024-04-24T01:55:18.215251+0000","last_active":"2024-04-24T02:00:22.277576+0000","last_peered":"2024-04-24T02:00:22.277576+0000","last_clean":"2024-04-24T02:00:22.277576+0000","last_became_active":"2024-04-24T01:55:18.206605+0000","last_became_peered":"2024-04-24T01:55:18.206605+0000","last_unstale":"2024-04-24T02:00:22.277576+0000","last_undegraded":"2024-04-24T02:00:22.277576+0000","last_fullsized":"2024-04-24T02:00:22.277576+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"301'174","reported_seq":1079,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-24T02:00:18.368171+0000","last_change":"2024-04-24T02:00:18.407398+0000","last_active":"2024-04-24T02:00:18.368171+0000","last_peered":"2024-04-24T02:00:18.368171+0000","last_clean":"2024-04-24T02:00:18.368171+0000","last_became_active":"2024-04-24T01:55:30.401366+0000","last_became_peered":"2024-04-24T01:55:30.401366+0000","last_unstale":"2024-04-24T02:00:18.368171+0000","last_undegraded":"2024-04-24T02:00:18.368171+0000","last_fullsized":"2024-04-24T02:00:18.368171+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":174,"log_dups_size":0,"ondisk_log_size":174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":13,"snaptrim_duration":0,"stat_sum":{"num_bytes":2117814,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":141,"num_read_kb":111398,"num_write":70,"num_write_kb":17540,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":777,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-04-24T02:00:19.162776+0000","last_change":"2024-04-24T01:55:18.230893+0000","last_active":"2024-04-24T02:00:19.162776+0000","last_peered":"2024-04-24T02:00:19.162776+0000","last_clean":"2024-04-24T02:00:19.162776+0000","last_became_active":"2024-04-24T01:55:18.225188+0000","last_became_peered":"2024-04-24T01:55:18.225188+0000","last_unstale":"2024-04-24T02:00:19.162776+0000","last_undegraded":"2024-04-24T02:00:19.162776+0000","last_fullsized":"2024-04-24T02:00:19.162776+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"301'272","reported_seq":1127,"reported_epoch":301,"state":"active+clean+snaptrim","last_fresh":"2024-04-24T02:00:18.291057+0000","last_change":"2024-04-24T02:00:18.166573+0000","last_active":"2024-04-24T02:00:18.291057+0000","last_peered":"2024-04-24T02:00:18.291057+0000","last_clean":"2024-04-24T02:00:18.291057+0000","last_became_active":"2024-04-24T01:55:30.343318+0000","last_became_peered":"2024-04-24T01:55:30.343318+0000","last_unstale":"2024-04-24T02:00:18.291057+0000","last_undegraded":"2024-04-24T02:00:18.291057+0000","last_fullsized":"2024-04-24T02:00:18.291057+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":272,"log_dups_size":0,"ondisk_log_size":272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":25,"snaptrim_duration":0,"stat_sum":{"num_bytes":7680997,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":148,"num_read_kb":122483,"num_write":125,"num_write_kb":35132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"305'399","reported_seq":1265,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-04-24T02:00:22.343208+0000","last_change":"2024-04-24T02:00:22.343208+0000","last_active":"2024-04-24T02:00:22.343208+0000","last_peered":"2024-04-24T02:00:22.343208+0000","last_clean":"2024-04-24T02:00:22.343208+0000","last_became_active":"2024-04-24T01:55:30.345308+0000","last_became_peered":"2024-04-24T01:55:30.345308+0000","last_unstale":"2024-04-24T02:00:22.343208+0000","last_undegraded":"2024-04-24T02:00:22.343208+0000","last_fullsized":"2024-04-24T02:00:22.343208+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":399,"log_dups_size":0,"ondisk_log_size":399,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":32,"snaptrim_duration":0,"stat_sum":{"num_bytes":14353574,"num_objects":6,"num_object_clones":2,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":201,"num_read_kb":130752,"num_write":196,"num_write_kb":52446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":746,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-24T02:00:18.172192+0000","last_change":"2024-04-24T01:55:18.224238+0000","last_active":"2024-04-24T02:00:18.172192+0000","last_peered":"2024-04-24T02:00:18.172192+0000","last_clean":"2024-04-24T02:00:18.172192+0000","last_became_active":"2024-04-24T01:55:18.219131+0000","last_became_peered":"2024-04-24T01:55:18.219131+0000","last_unstale":"2024-04-24T02:00:18.172192+0000","last_undegraded":"2024-04-24T02:00:18.172192+0000","last_fullsized":"2024-04-24T02:00:18.172192+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"291'79","reported_seq":955,"reported_epoch":301,"state":"active+clean+snaptrim","last_fresh":"2024-04-24T02:00:18.175982+0000","last_change":"2024-04-24T02:00:18.175520+0000","last_active":"2024-04-24T02:00:18.175982+0000","last_peered":"2024-04-24T02:00:18.175982+0000","last_clean":"2024-04-24T02:00:18.175982+0000","last_became_active":"2024-04-24T01:55:30.319958+0000","last_became_peered":"2024-04-24T01:55:30.319958+0000","last_unstale":"2024-04-24T02:00:18.175982+0000","last_undegraded":"2024-04-24T02:00:18.175982+0000","last_fullsized":"2024-04-24T02:00:18.175982+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0,"stat_sum":{"num_bytes":3882516,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":41,"num_read_kb":36474,"num_write":51,"num_write_kb":14818,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":766,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-24T02:00:18.360260+0000","last_change":"2024-04-24T01:55:18.235763+0000","last_active":"2024-04-24T02:00:18.360260+0000","last_peered":"2024-04-24T02:00:18.360260+0000","last_clean":"2024-04-24T02:00:18.360260+0000","last_became_active":"2024-04-24T01:55:18.229175+0000","last_became_peered":"2024-04-24T01:55:18.229175+0000","last_unstale":"2024-04-24T02:00:18.360260+0000","last_undegraded":"2024-04-24T02:00:18.360260+0000","last_fullsized":"2024-04-24T02:00:18.360260+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"281'149","reported_seq":1032,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-24T02:00:18.347863+0000","last_change":"2024-04-24T02:00:18.359384+0000","last_active":"2024-04-24T02:00:18.347863+0000","last_peered":"2024-04-24T02:00:18.347863+0000","last_clean":"2024-04-24T02:00:18.347863+0000","last_became_active":"2024-04-24T01:55:30.372690+0000","last_became_peered":"2024-04-24T01:55:30.372690+0000","last_unstale":"2024-04-24T02:00:18.347863+0000","last_undegraded":"2024-04-24T02:00:18.347863+0000","last_fullsized":"2024-04-24T02:00:18.347863+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":13,"snaptrim_duration":0,"stat_sum":{"num_bytes":6857454,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":74,"num_read_kb":56402,"num_write":69,"num_write_kb":19977,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":746,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-24T02:00:18.181741+0000","last_change":"2024-04-24T01:55:18.278351+0000","last_active":"2024-04-24T02:00:18.181741+0000","last_peered":"2024-04-24T02:00:18.181741+0000","last_clean":"2024-04-24T02:00:18.181741+0000","last_became_active":"2024-04-24T01:55:18.273668+0000","last_became_peered":"2024-04-24T01:55:18.273668+0000","last_unstale":"2024-04-24T02:00:18.181741+0000","last_undegraded":"2024-04-24T02:00:18.181741+0000","last_fullsized":"2024-04-24T02:00:18.181741+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"301'335","reported_seq":1219,"reported_epoch":301,"state":"active+clean+snaptrim","last_fresh":"2024-04-24T02:00:18.287829+0000","last_change":"2024-04-24T02:00:18.179297+0000","last_active":"2024-04-24T02:00:18.287829+0000","last_peered":"2024-04-24T02:00:18.287829+0000","last_clean":"2024-04-24T02:00:18.287829+0000","last_became_active":"2024-04-24T01:55:30.363123+0000","last_became_peered":"2024-04-24T01:55:30.363123+0000","last_unstale":"2024-04-24T02:00:18.287829+0000","last_undegraded":"2024-04-24T02:00:18.287829+0000","last_fullsized":"2024-04-24T02:00:18.287829+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":335,"log_dups_size":0,"ondisk_log_size":335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":35,"snaptrim_duration":0,"stat_sum":{"num_bytes":10958461,"num_objects":7,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":227,"num_read_kb":196496,"num_write":179,"num_write_kb":48260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":762,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-04-24T02:00:22.293729+0000","last_change":"2024-04-24T01:55:18.241117+0000","last_active":"2024-04-24T02:00:22.293729+0000","last_peered":"2024-04-24T02:00:22.293729+0000","last_clean":"2024-04-24T02:00:22.293729+0000","last_became_active":"2024-04-24T01:55:18.234313+0000","last_became_peered":"2024-04-24T01:55:18.234313+0000","last_unstale":"2024-04-24T02:00:22.293729+0000","last_undegraded":"2024-04-24T02:00:22.293729+0000","last_fullsized":"2024-04-24T02:00:22.293729+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":94626545,"num_objects":57,"num_object_clones":19,"num_object_copies":171,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":2262,"num_read_kb":1654407,"num_write":2052,"num_write_kb":558340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262291456,"data_stored":275064115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4121,"ondisk_log_size":4121,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":1}],"osd_stats":[{"osd":3,"up_from":10,"seq":42949673264,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":110980,"kb_used_data":75500,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94260860,"statfs":{"total":96636764160,"available":96523120640,"internally_reserved":0,"allocated":77312000,"data_stored":81715418,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673264,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":98868,"kb_used_data":63508,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94272972,"statfs":{"total":96636764160,"available":96535523328,"internally_reserved":0,"allocated":65032192,"data_stored":67611254,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705969,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":86260,"kb_used_data":50900,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94285580,"statfs":{"total":96636764160,"available":96548433920,"internally_reserved":0,"allocated":52121600,"data_stored":54197479,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705973,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":110140,"kb_used_data":74892,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94261700,"statfs":{"total":96636764160,"available":96523980800,"internally_reserved":0,"allocated":76689408,"data_stored":78256868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49811456,"data_stored":52602602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73527296,"data_stored":75528652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":63275008,"data_stored":66387083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":75677696,"data_stored":80545778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T02:00:24.488 INFO:tasks.rados.rados.0.smithi079.stdout:1285: finishing write tid 1 to smithi07929762-30 2024-04-24T02:00:24.488 INFO:tasks.rados.rados.0.smithi079.stdout:1285: finishing write tid 2 to smithi07929762-30 2024-04-24T02:00:24.489 INFO:tasks.rados.rados.0.smithi079.stdout:1285: finishing write tid 3 to smithi07929762-30 2024-04-24T02:00:24.489 INFO:tasks.rados.rados.0.smithi079.stdout:1285: finishing write tid 4 to smithi07929762-30 2024-04-24T02:00:24.489 INFO:tasks.rados.rados.0.smithi079.stdout:1285: oid 30 updating version 0 to 166 2024-04-24T02:00:24.489 INFO:tasks.rados.rados.0.smithi079.stdout:1285: oid 30 version 166 is already newer than 166 2024-04-24T02:00:24.489 INFO:tasks.rados.rados.0.smithi079.stdout:1285: oid 30 version 166 is already newer than 164 2024-04-24T02:00:24.489 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 30 v 166 (ObjNum 626 snap 144 seq_num 626) dirty exists 2024-04-24T02:00:24.489 INFO:tasks.rados.rados.0.smithi079.stdout:1285: left oid 30 (ObjNum 626 snap 144 seq_num 626) 2024-04-24T02:00:24.489 INFO:tasks.rados.rados.0.smithi079.stdout:1284: done (2 left) 2024-04-24T02:00:24.489 INFO:tasks.rados.rados.0.smithi079.stdout:1285: done (1 left) 2024-04-24T02:00:24.489 INFO:tasks.rados.rados.0.smithi079.stdout:1286: done (0 left) 2024-04-24T02:00:24.489 INFO:tasks.rados.rados.0.smithi079.stdout:1287: write (excl) oid 31 current snap is 144 2024-04-24T02:00:24.489 INFO:tasks.rados.rados.0.smithi079.stdout:1287: seq_num 628 ranges {554796=797318,1949280=526888} 2024-04-24T02:00:24.495 INFO:tasks.rados.rados.0.smithi079.stdout:1287: writing smithi07929762-31 from 554796 to 1352114 tid 1 2024-04-24T02:00:24.502 INFO:tasks.rados.rados.0.smithi079.stdout:1287: writing smithi07929762-31 from 1949280 to 2476168 tid 2 2024-04-24T02:00:24.502 INFO:tasks.rados.rados.0.smithi079.stdout:1288: snap_create 2024-04-24T02:00:24.524 INFO:tasks.rados.rados.0.smithi079.stdout:1287: finishing write tid 1 to smithi07929762-31 2024-04-24T02:00:24.539 INFO:tasks.rados.rados.0.smithi079.stdout:1287: finishing write tid 2 to smithi07929762-31 2024-04-24T02:00:24.542 INFO:tasks.rados.rados.0.smithi079.stdout:1287: finishing write tid 3 to smithi07929762-31 2024-04-24T02:00:24.543 INFO:tasks.rados.rados.0.smithi079.stdout:1287: finishing write tid 4 to smithi07929762-31 2024-04-24T02:00:24.543 INFO:tasks.rados.rados.0.smithi079.stdout:1287: oid 31 updating version 0 to 599 2024-04-24T02:00:24.543 INFO:tasks.rados.rados.0.smithi079.stdout:1287: oid 31 updating version 599 to 601 2024-04-24T02:00:24.543 INFO:tasks.rados.rados.0.smithi079.stdout:1287: oid 31 version 601 is already newer than 601 2024-04-24T02:00:24.544 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 31 v 601 (ObjNum 628 snap 144 seq_num 628) dirty exists 2024-04-24T02:00:24.544 INFO:tasks.rados.rados.0.smithi079.stdout:1287: left oid 31 (ObjNum 628 snap 144 seq_num 628) 2024-04-24T02:00:24.567 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T02:00:24.671 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:24.669+0000 7f596802d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:24.675 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:24.674+0000 7f596802d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:25.163 INFO:tasks.rados.rados.0.smithi079.stdout:1287: done (1 left) 2024-04-24T02:00:25.164 INFO:tasks.rados.rados.0.smithi079.stdout:1288: done (0 left) 2024-04-24T02:00:25.164 INFO:tasks.rados.rados.0.smithi079.stdout:1289: delete oid 40 current snap is 145 2024-04-24T02:00:25.237 INFO:tasks.rados.rados.0.smithi079.stdout:1289: done (0 left) 2024-04-24T02:00:25.237 INFO:tasks.rados.rados.0.smithi079.stdout:1290: write (excl) oid 27 current snap is 145 2024-04-24T02:00:25.237 INFO:tasks.rados.rados.0.smithi079.stdout:1290: seq_num 630 ranges {588410=547816,1666244=414204,2484603=103807} 2024-04-24T02:00:25.242 INFO:tasks.rados.rados.0.smithi079.stdout:1290: writing smithi07929762-27 from 588410 to 1136226 tid 1 2024-04-24T02:00:25.248 INFO:tasks.rados.rados.0.smithi079.stdout:1290: writing smithi07929762-27 from 1666244 to 2080448 tid 2 2024-04-24T02:00:25.250 INFO:tasks.rados.rados.0.smithi079.stdout:1290: writing smithi07929762-27 from 2484603 to 2588410 tid 3 2024-04-24T02:00:25.251 INFO:tasks.rados.rados.0.smithi079.stdout:1291: delete oid 49 current snap is 145 2024-04-24T02:00:25.288 INFO:tasks.rados.rados.0.smithi079.stdout:1291: done (1 left) 2024-04-24T02:00:25.288 INFO:tasks.rados.rados.0.smithi079.stdout:1292: delete oid 21 current snap is 145 2024-04-24T02:00:25.364 INFO:tasks.rados.rados.0.smithi079.stdout:1290: finishing write tid 1 to smithi07929762-27 2024-04-24T02:00:25.395 INFO:tasks.rados.rados.0.smithi079.stdout:1290: finishing write tid 2 to smithi07929762-27 2024-04-24T02:00:25.398 INFO:tasks.rados.rados.0.smithi079.stdout:1292: done (1 left) 2024-04-24T02:00:25.398 INFO:tasks.rados.rados.0.smithi079.stdout:1293: snap_create 2024-04-24T02:00:25.399 INFO:tasks.rados.rados.0.smithi079.stdout:1290: finishing write tid 3 to smithi07929762-27 2024-04-24T02:00:25.400 INFO:tasks.rados.rados.0.smithi079.stdout:1290: finishing write tid 4 to smithi07929762-27 2024-04-24T02:00:25.400 INFO:tasks.rados.rados.0.smithi079.stdout:1290: finishing write tid 5 to smithi07929762-27 2024-04-24T02:00:25.400 INFO:tasks.rados.rados.0.smithi079.stdout:1290: oid 27 updating version 0 to 123 2024-04-24T02:00:25.400 INFO:tasks.rados.rados.0.smithi079.stdout:1290: oid 27 version 123 is already newer than 120 2024-04-24T02:00:25.400 INFO:tasks.rados.rados.0.smithi079.stdout:1290: oid 27 version 123 is already newer than 123 2024-04-24T02:00:25.400 INFO:tasks.rados.rados.0.smithi079.stdout:1290: oid 27 version 123 is already newer than 123 2024-04-24T02:00:25.401 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 27 v 123 (ObjNum 630 snap 145 seq_num 630) dirty exists 2024-04-24T02:00:25.401 INFO:tasks.rados.rados.0.smithi079.stdout:1290: left oid 27 (ObjNum 630 snap 145 seq_num 630) 2024-04-24T02:00:26.164 INFO:tasks.rados.rados.0.smithi079.stdout:1290: done (1 left) 2024-04-24T02:00:26.164 INFO:tasks.rados.rados.0.smithi079.stdout:1293: done (0 left) 2024-04-24T02:00:26.164 INFO:tasks.rados.rados.0.smithi079.stdout:1294: snap_create 2024-04-24T02:00:26.767 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T02:00:26.873 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:26.871+0000 7f80a599d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:26.877 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:26.876+0000 7f80a599d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:27.167 INFO:tasks.rados.rados.0.smithi079.stdout:1294: done (0 left) 2024-04-24T02:00:27.167 INFO:tasks.rados.rados.0.smithi079.stdout:1295: read oid 20 snap -1 2024-04-24T02:00:27.167 INFO:tasks.rados.rados.0.smithi079.stdout:1295: expect (ObjNum 565 snap 130 seq_num 565) 2024-04-24T02:00:27.167 INFO:tasks.rados.rados.0.smithi079.stdout:1296: snap_remove snap 145 2024-04-24T02:00:28.169 INFO:tasks.rados.rados.0.smithi079.stdout:1296: done (1 left) 2024-04-24T02:00:28.169 INFO:tasks.rados.rados.0.smithi079.stdout:1297: snap_create 2024-04-24T02:00:28.169 INFO:tasks.rados.rados.0.smithi079.stdout:1295: expect (ObjNum 565 snap 130 seq_num 565) 2024-04-24T02:00:28.971 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T02:00:29.079 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:29.076+0000 7f0171d5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:29.082 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:29.082+0000 7f0171d5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:29.174 INFO:tasks.rados.rados.0.smithi079.stdout:1295: done (1 left) 2024-04-24T02:00:29.175 INFO:tasks.rados.rados.0.smithi079.stdout:1297: done (0 left) 2024-04-24T02:00:29.175 INFO:tasks.rados.rados.0.smithi079.stdout:1298: delete oid 50 current snap is 148 2024-04-24T02:00:29.219 INFO:tasks.rados.rados.0.smithi079.stdout:1298: done (0 left) 2024-04-24T02:00:29.219 INFO:tasks.rados.rados.0.smithi079.stdout:1299: write (excl) oid 1 current snap is 148 2024-04-24T02:00:29.219 INFO:tasks.rados.rados.0.smithi079.stdout:1299: seq_num 634 ranges {655638=565165,1900172=421542} 2024-04-24T02:00:29.225 INFO:tasks.rados.rados.0.smithi079.stdout:1299: writing smithi07929762-1 from 655638 to 1220803 tid 1 2024-04-24T02:00:29.232 INFO:tasks.rados.rados.0.smithi079.stdout:1299: writing smithi07929762-1 from 1900172 to 2321714 tid 2 2024-04-24T02:00:29.232 INFO:tasks.rados.rados.0.smithi079.stdout:1300: snap_create 2024-04-24T02:00:29.252 INFO:tasks.rados.rados.0.smithi079.stdout:1299: finishing write tid 1 to smithi07929762-1 2024-04-24T02:00:29.267 INFO:tasks.rados.rados.0.smithi079.stdout:1299: finishing write tid 2 to smithi07929762-1 2024-04-24T02:00:29.270 INFO:tasks.rados.rados.0.smithi079.stdout:1299: finishing write tid 3 to smithi07929762-1 2024-04-24T02:00:29.272 INFO:tasks.rados.rados.0.smithi079.stdout:1299: finishing write tid 4 to smithi07929762-1 2024-04-24T02:00:29.272 INFO:tasks.rados.rados.0.smithi079.stdout:1299: oid 1 updating version 0 to 608 2024-04-24T02:00:29.272 INFO:tasks.rados.rados.0.smithi079.stdout:1299: oid 1 version 608 is already newer than 606 2024-04-24T02:00:29.273 INFO:tasks.rados.rados.0.smithi079.stdout:1299: oid 1 version 608 is already newer than 608 2024-04-24T02:00:29.273 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 1 v 608 (ObjNum 634 snap 148 seq_num 634) dirty exists 2024-04-24T02:00:29.273 INFO:tasks.rados.rados.0.smithi079.stdout:1299: left oid 1 (ObjNum 634 snap 148 seq_num 634) 2024-04-24T02:00:29.399 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-04-24T02:00:29.399 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-04-24T02:00:29.399 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-04-24T02:00:29.399 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-04-24T02:00:29.506 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:29.501+0000 7fd914206640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:29.510 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:29.510+0000 7fd914206640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:29.525 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T02:00:29.525 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T02:00:29.525 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T02:00:29.525 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T02:00:29.525 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T02:00:29.525 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T02:00:29.525 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T02:00:29.525 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T02:00:29.526 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T02:00:29.526 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T02:00:29.526 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T02:00:29.526 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T02:00:29.528 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T02:00:29.528 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T02:00:29.528 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-04-24T02:00:29.636 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:29.630+0000 7ff88e46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:29.639 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:29.638+0000 7ff88e46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:29.653 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T02:00:29.653 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T02:00:29.654 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T02:00:29.654 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T02:00:29.654 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T02:00:29.654 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T02:00:29.654 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T02:00:29.654 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T02:00:29.654 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T02:00:29.654 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T02:00:29.654 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T02:00:29.654 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T02:00:29.656 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T02:00:29.657 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T02:00:29.657 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-04-24T02:00:29.766 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:29.761+0000 7fbe80d5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:29.770 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:29.769+0000 7fbe80d5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:29.788 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T02:00:29.788 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T02:00:29.788 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T02:00:29.788 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T02:00:29.788 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T02:00:29.788 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T02:00:29.788 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T02:00:29.788 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T02:00:29.788 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T02:00:29.788 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T02:00:29.788 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T02:00:29.789 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T02:00:29.790 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T02:00:29.791 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T02:00:29.791 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-04-24T02:00:29.898 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:29.892+0000 7f5c5ba6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:29.902 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:29.901+0000 7f5c5ba6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:29.919 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T02:00:29.919 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T02:00:29.919 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T02:00:29.919 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T02:00:29.919 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T02:00:29.919 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T02:00:29.919 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T02:00:29.919 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T02:00:29.919 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T02:00:29.919 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T02:00:29.919 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T02:00:29.920 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T02:00:29.922 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T02:00:29.922 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T02:00:30.178 INFO:tasks.rados.rados.0.smithi079.stdout:1299: done (1 left) 2024-04-24T02:00:30.178 INFO:tasks.rados.rados.0.smithi079.stdout:1300: done (0 left) 2024-04-24T02:00:30.178 INFO:tasks.rados.rados.0.smithi079.stdout:1301: write (excl) oid 33 current snap is 149 2024-04-24T02:00:30.178 INFO:tasks.rados.rados.0.smithi079.stdout:1301: seq_num 635 ranges {672445=640414,1842301=681553} 2024-04-24T02:00:30.185 INFO:tasks.rados.rados.0.smithi079.stdout:1301: writing smithi07929762-33 from 672445 to 1312859 tid 1 2024-04-24T02:00:30.192 INFO:tasks.rados.rados.0.smithi079.stdout:1301: writing smithi07929762-33 from 1842301 to 2523854 tid 2 2024-04-24T02:00:30.193 INFO:tasks.rados.rados.0.smithi079.stdout:1302: snap_remove snap 144 2024-04-24T02:00:31.181 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T02:00:31.183 INFO:tasks.rados.rados.0.smithi079.stdout:1302: done (1 left) 2024-04-24T02:00:31.183 INFO:tasks.rados.rados.0.smithi079.stdout:1303: delete oid 41 current snap is 149 2024-04-24T02:00:31.183 INFO:tasks.rados.rados.0.smithi079.stdout:1301: finishing write tid 1 to smithi07929762-33 2024-04-24T02:00:31.183 INFO:tasks.rados.rados.0.smithi079.stdout:1301: finishing write tid 2 to smithi07929762-33 2024-04-24T02:00:31.183 INFO:tasks.rados.rados.0.smithi079.stdout:1301: finishing write tid 3 to smithi07929762-33 2024-04-24T02:00:31.183 INFO:tasks.rados.rados.0.smithi079.stdout:1301: finishing write tid 4 to smithi07929762-33 2024-04-24T02:00:31.183 INFO:tasks.rados.rados.0.smithi079.stdout:1301: oid 33 updating version 0 to 409 2024-04-24T02:00:31.183 INFO:tasks.rados.rados.0.smithi079.stdout:1301: oid 33 version 409 is already newer than 407 2024-04-24T02:00:31.183 INFO:tasks.rados.rados.0.smithi079.stdout:1301: oid 33 version 409 is already newer than 409 2024-04-24T02:00:31.183 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 33 v 409 (ObjNum 635 snap 149 seq_num 635) dirty exists 2024-04-24T02:00:31.184 INFO:tasks.rados.rados.0.smithi079.stdout:1301: left oid 33 (ObjNum 635 snap 149 seq_num 635) 2024-04-24T02:00:31.256 INFO:tasks.rados.rados.0.smithi079.stdout:1301: done (1 left) 2024-04-24T02:00:31.256 INFO:tasks.rados.rados.0.smithi079.stdout:1303: done (0 left) 2024-04-24T02:00:31.256 INFO:tasks.rados.rados.0.smithi079.stdout:1304: snap_create 2024-04-24T02:00:31.301 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:31.299+0000 7fa86a42d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:31.305 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:31.304+0000 7fa86a42d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:32.182 INFO:tasks.rados.rados.0.smithi079.stdout:1304: done (0 left) 2024-04-24T02:00:32.182 INFO:tasks.rados.rados.0.smithi079.stdout:1305: delete oid 46 current snap is 150 2024-04-24T02:00:32.258 INFO:tasks.rados.rados.0.smithi079.stdout:1305: done (0 left) 2024-04-24T02:00:32.258 INFO:tasks.rados.rados.0.smithi079.stdout:1306: delete oid 16 current snap is 150 2024-04-24T02:00:32.268 INFO:tasks.rados.rados.0.smithi079.stdout:1306: done (0 left) 2024-04-24T02:00:32.268 INFO:tasks.rados.rados.0.smithi079.stdout:1307: write (excl) oid 24 current snap is 150 2024-04-24T02:00:32.268 INFO:tasks.rados.rados.0.smithi079.stdout:1307: seq_num 639 ranges {739673=657763,2076229=663444} 2024-04-24T02:00:32.275 INFO:tasks.rados.rados.0.smithi079.stdout:1307: writing smithi07929762-24 from 739673 to 1397436 tid 1 2024-04-24T02:00:32.284 INFO:tasks.rados.rados.0.smithi079.stdout:1307: writing smithi07929762-24 from 2076229 to 2739673 tid 2 2024-04-24T02:00:32.284 INFO:tasks.rados.rados.0.smithi079.stdout:1308: read oid 40 snap -1 2024-04-24T02:00:32.284 INFO:tasks.rados.rados.0.smithi079.stdout:1308: expect deleted 2024-04-24T02:00:32.284 INFO:tasks.rados.rados.0.smithi079.stdout:1309: write oid 30 current snap is 150 2024-04-24T02:00:32.284 INFO:tasks.rados.rados.0.smithi079.stdout:1309: seq_num 640 ranges {756480=733012,2134711=432549} 2024-04-24T02:00:32.292 INFO:tasks.rados.rados.0.smithi079.stdout:1309: writing smithi07929762-30 from 756480 to 1489492 tid 1 2024-04-24T02:00:32.298 INFO:tasks.rados.rados.0.smithi079.stdout:1309: writing smithi07929762-30 from 2134711 to 2567260 tid 2 2024-04-24T02:00:32.298 INFO:tasks.rados.rados.0.smithi079.stdout:1310: write (excl) oid 23 current snap is 150 2024-04-24T02:00:32.298 INFO:tasks.rados.rados.0.smithi079.stdout:1310: seq_num 641 ranges {773287=408261,1793193=576207} 2024-04-24T02:00:32.304 INFO:tasks.rados.rados.0.smithi079.stdout:1310: writing smithi07929762-23 from 773287 to 1181548 tid 1 2024-04-24T02:00:32.313 INFO:tasks.rados.rados.0.smithi079.stdout:1310: writing smithi07929762-23 from 1793193 to 2369400 tid 2 2024-04-24T02:00:32.313 INFO:tasks.rados.rados.0.smithi079.stdout:1311: snap_create 2024-04-24T02:00:32.350 INFO:tasks.rados.rados.0.smithi079.stdout:1310: finishing write tid 1 to smithi07929762-23 2024-04-24T02:00:32.374 INFO:tasks.rados.rados.0.smithi079.stdout:1307: finishing write tid 1 to smithi07929762-24 2024-04-24T02:00:32.392 INFO:tasks.rados.rados.0.smithi079.stdout:1310: finishing write tid 2 to smithi07929762-23 2024-04-24T02:00:32.399 INFO:tasks.rados.rados.0.smithi079.stdout:1309: finishing write tid 1 to smithi07929762-30 2024-04-24T02:00:32.409 INFO:tasks.rados.rados.0.smithi079.stdout:1310: finishing write tid 3 to smithi07929762-23 2024-04-24T02:00:32.409 INFO:tasks.rados.rados.0.smithi079.stdout:1309: finishing write tid 2 to smithi07929762-30 2024-04-24T02:00:32.409 INFO:tasks.rados.rados.0.smithi079.stdout:1310: finishing write tid 4 to smithi07929762-23 2024-04-24T02:00:32.409 INFO:tasks.rados.rados.0.smithi079.stdout:1310: oid 23 updating version 0 to 117 2024-04-24T02:00:32.410 INFO:tasks.rados.rados.0.smithi079.stdout:1310: oid 23 version 117 is already newer than 117 2024-04-24T02:00:32.410 INFO:tasks.rados.rados.0.smithi079.stdout:1310: oid 23 version 117 is already newer than 115 2024-04-24T02:00:32.410 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 23 v 117 (ObjNum 641 snap 150 seq_num 641) dirty exists 2024-04-24T02:00:32.410 INFO:tasks.rados.rados.0.smithi079.stdout:1310: left oid 23 (ObjNum 641 snap 150 seq_num 641) 2024-04-24T02:00:32.413 INFO:tasks.rados.rados.0.smithi079.stdout:1307: finishing write tid 2 to smithi07929762-24 2024-04-24T02:00:32.414 INFO:tasks.rados.rados.0.smithi079.stdout:1309: finishing write tid 3 to smithi07929762-30 2024-04-24T02:00:32.416 INFO:tasks.rados.rados.0.smithi079.stdout:1309: finishing write tid 4 to smithi07929762-30 2024-04-24T02:00:32.417 INFO:tasks.rados.rados.0.smithi079.stdout:1309: oid 30 updating version 0 to 170 2024-04-24T02:00:32.417 INFO:tasks.rados.rados.0.smithi079.stdout:1309: oid 30 version 170 is already newer than 170 2024-04-24T02:00:32.417 INFO:tasks.rados.rados.0.smithi079.stdout:1309: oid 30 version 170 is already newer than 170 2024-04-24T02:00:32.417 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 30 v 170 (ObjNum 640 snap 150 seq_num 640) dirty exists 2024-04-24T02:00:32.417 INFO:tasks.rados.rados.0.smithi079.stdout:1309: left oid 30 (ObjNum 640 snap 150 seq_num 640) 2024-04-24T02:00:32.417 INFO:tasks.rados.rados.0.smithi079.stdout:1307: finishing write tid 3 to smithi07929762-24 2024-04-24T02:00:32.419 INFO:tasks.rados.rados.0.smithi079.stdout:1307: finishing write tid 4 to smithi07929762-24 2024-04-24T02:00:32.419 INFO:tasks.rados.rados.0.smithi079.stdout:1307: oid 24 updating version 0 to 343 2024-04-24T02:00:32.419 INFO:tasks.rados.rados.0.smithi079.stdout:1307: oid 24 version 343 is already newer than 341 2024-04-24T02:00:32.419 INFO:tasks.rados.rados.0.smithi079.stdout:1307: oid 24 version 343 is already newer than 343 2024-04-24T02:00:32.419 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 24 v 343 (ObjNum 639 snap 150 seq_num 639) dirty exists 2024-04-24T02:00:32.419 INFO:tasks.rados.rados.0.smithi079.stdout:1307: left oid 24 (ObjNum 639 snap 150 seq_num 639) 2024-04-24T02:00:33.188 INFO:tasks.rados.rados.0.smithi079.stdout:1307: done (4 left) 2024-04-24T02:00:33.189 INFO:tasks.rados.rados.0.smithi079.stdout:1308: done (3 left) 2024-04-24T02:00:33.189 INFO:tasks.rados.rados.0.smithi079.stdout:1309: done (2 left) 2024-04-24T02:00:33.189 INFO:tasks.rados.rados.0.smithi079.stdout:1310: done (1 left) 2024-04-24T02:00:33.189 INFO:tasks.rados.rados.0.smithi079.stdout:1311: done (0 left) 2024-04-24T02:00:33.189 INFO:tasks.rados.rados.0.smithi079.stdout:1312: read oid 1 snap 147 2024-04-24T02:00:33.189 INFO:tasks.rados.rados.0.smithi079.stdout:1312: expect deleted 2024-04-24T02:00:33.189 INFO:tasks.rados.rados.0.smithi079.stdout:1313: snap_remove snap 146 2024-04-24T02:00:33.398 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T02:00:33.505 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:33.504+0000 7fca8766d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:33.509 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:33.508+0000 7fca8766d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:34.190 INFO:tasks.rados.rados.0.smithi079.stdout:1313: done (1 left) 2024-04-24T02:00:34.190 INFO:tasks.rados.rados.0.smithi079.stdout:1314: write oid 39 current snap is 151 2024-04-24T02:00:34.190 INFO:tasks.rados.rados.0.smithi079.stdout:1314: seq_num 642 ranges {790094=483510,1851675=436218} 2024-04-24T02:00:34.196 INFO:tasks.rados.rados.0.smithi079.stdout:1314: writing smithi07929762-39 from 790094 to 1273604 tid 1 2024-04-24T02:00:34.202 INFO:tasks.rados.rados.0.smithi079.stdout:1314: writing smithi07929762-39 from 1851675 to 2287893 tid 2 2024-04-24T02:00:34.202 INFO:tasks.rados.rados.0.smithi079.stdout:1315: delete oid 34 current snap is 151 2024-04-24T02:00:34.246 INFO:tasks.rados.rados.0.smithi079.stdout:1312: done (2 left) 2024-04-24T02:00:34.246 INFO:tasks.rados.rados.0.smithi079.stdout:1315: done (1 left) 2024-04-24T02:00:34.246 INFO:tasks.rados.rados.0.smithi079.stdout:1316: write (excl) oid 10 current snap is 151 2024-04-24T02:00:34.246 INFO:tasks.rados.rados.0.smithi079.stdout:1316: seq_num 644 ranges {423708=634008,1852286=439887,2752172=71536} 2024-04-24T02:00:34.252 INFO:tasks.rados.rados.0.smithi079.stdout:1316: writing smithi07929762-10 from 423708 to 1057716 tid 1 2024-04-24T02:00:34.260 INFO:tasks.rados.rados.0.smithi079.stdout:1316: writing smithi07929762-10 from 1852286 to 2292173 tid 2 2024-04-24T02:00:34.263 INFO:tasks.rados.rados.0.smithi079.stdout:1316: writing smithi07929762-10 from 2752172 to 2823708 tid 3 2024-04-24T02:00:34.263 INFO:tasks.rados.rados.0.smithi079.stdout:1317: read oid 28 snap -1 2024-04-24T02:00:34.263 INFO:tasks.rados.rados.0.smithi079.stdout:1317: expect deleted 2024-04-24T02:00:34.263 INFO:tasks.rados.rados.0.smithi079.stdout:1318: write (excl) oid 3 current snap is 151 2024-04-24T02:00:34.263 INFO:tasks.rados.rados.0.smithi079.stdout:1318: seq_num 645 ranges {440515=709257,1910768=583545} 2024-04-24T02:00:34.270 INFO:tasks.rados.rados.0.smithi079.stdout:1318: writing smithi07929762-3 from 440515 to 1149772 tid 1 2024-04-24T02:00:34.278 INFO:tasks.rados.rados.0.smithi079.stdout:1318: writing smithi07929762-3 from 1910768 to 2494313 tid 2 2024-04-24T02:00:34.278 INFO:tasks.rados.rados.0.smithi079.stdout:1319: write oid 36 current snap is 151 2024-04-24T02:00:34.278 INFO:tasks.rados.rados.0.smithi079.stdout:1319: seq_num 646 ranges {457322=784506,1969250=443556} 2024-04-24T02:00:34.285 INFO:tasks.rados.rados.0.smithi079.stdout:1319: writing smithi07929762-36 from 457322 to 1241828 tid 1 2024-04-24T02:00:34.292 INFO:tasks.rados.rados.0.smithi079.stdout:1319: writing smithi07929762-36 from 1969250 to 2412806 tid 2 2024-04-24T02:00:34.292 INFO:tasks.rados.rados.0.smithi079.stdout:1320: read oid 33 snap -1 2024-04-24T02:00:34.292 INFO:tasks.rados.rados.0.smithi079.stdout:1320: expect (ObjNum 635 snap 149 seq_num 635) 2024-04-24T02:00:34.293 INFO:tasks.rados.rados.0.smithi079.stdout:1321: snap_remove snap 150 2024-04-24T02:00:34.923 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-04-24T02:00:34.923 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-04-24T02:00:34.923 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T02:00:35.030 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:35.027+0000 7f59f986d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:35.034 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:35.034+0000 7f59f986d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:35.195 INFO:tasks.rados.rados.0.smithi079.stdout:1321: done (6 left) 2024-04-24T02:00:35.195 INFO:tasks.rados.rados.0.smithi079.stdout:1322: snap_create 2024-04-24T02:00:35.195 INFO:tasks.rados.rados.0.smithi079.stdout:1318: finishing write tid 1 to smithi07929762-3 2024-04-24T02:00:35.195 INFO:tasks.rados.rados.0.smithi079.stdout:1314: finishing write tid 1 to smithi07929762-39 2024-04-24T02:00:35.195 INFO:tasks.rados.rados.0.smithi079.stdout:1318: finishing write tid 2 to smithi07929762-3 2024-04-24T02:00:35.195 INFO:tasks.rados.rados.0.smithi079.stdout:1318: finishing write tid 3 to smithi07929762-3 2024-04-24T02:00:35.195 INFO:tasks.rados.rados.0.smithi079.stdout:1314: finishing write tid 2 to smithi07929762-39 2024-04-24T02:00:35.195 INFO:tasks.rados.rados.0.smithi079.stdout:1318: finishing write tid 4 to smithi07929762-3 2024-04-24T02:00:35.196 INFO:tasks.rados.rados.0.smithi079.stdout:1318: oid 3 updating version 0 to 613 2024-04-24T02:00:35.196 INFO:tasks.rados.rados.0.smithi079.stdout:1318: oid 3 updating version 613 to 615 2024-04-24T02:00:35.196 INFO:tasks.rados.rados.0.smithi079.stdout:1318: oid 3 version 615 is already newer than 615 2024-04-24T02:00:35.196 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 3 v 615 (ObjNum 645 snap 151 seq_num 645) dirty exists 2024-04-24T02:00:35.196 INFO:tasks.rados.rados.0.smithi079.stdout:1318: left oid 3 (ObjNum 645 snap 151 seq_num 645) 2024-04-24T02:00:35.196 INFO:tasks.rados.rados.0.smithi079.stdout:1319: finishing write tid 1 to smithi07929762-36 2024-04-24T02:00:35.196 INFO:tasks.rados.rados.0.smithi079.stdout:1320: expect (ObjNum 635 snap 149 seq_num 635) 2024-04-24T02:00:35.220 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T02:00:35.220 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-24T02:00:35.232 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":470,"stamp":"2024-04-24T02:00:35.193826+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":89435602,"num_objects":39,"num_object_clones":5,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":2410,"num_read_kb":1662995,"num_write":2261,"num_write_kb":577559,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4369,"ondisk_log_size":4369,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":316092,"kb_used_data":174972,"kb_used_omap":0,"kb_used_meta":140544,"kb_avail":377171268,"statfs":{"total":386547056640,"available":386223378432,"internally_reserved":0,"allocated":179171328,"data_stored":183315372,"data_compressed":89889,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":143917056},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1092653,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":-1,"num_read":10,"num_read_kb":5,"num_write":21,"num_write_kb":5648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.013359"},"pg_stats":[{"pgid":"3.f","version":"307'242","reported_seq":1195,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-04-24T02:00:33.211585+0000","last_change":"2024-04-24T02:00:32.239316+0000","last_active":"2024-04-24T02:00:33.211585+0000","last_peered":"2024-04-24T02:00:33.211585+0000","last_clean":"2024-04-24T02:00:33.211585+0000","last_became_active":"2024-04-24T01:55:30.386726+0000","last_became_peered":"2024-04-24T01:55:30.386726+0000","last_unstale":"2024-04-24T02:00:33.211585+0000","last_undegraded":"2024-04-24T02:00:33.211585+0000","last_fullsized":"2024-04-24T02:00:33.211585+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":242,"log_dups_size":0,"ondisk_log_size":242,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":23,"snaptrim_duration":0,"stat_sum":{"num_bytes":4370084,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":172,"num_read_kb":152845,"num_write":111,"num_write_kb":32111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"308'358","reported_seq":1257,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-04-24T02:00:30.273895+0000","last_change":"2024-04-24T02:00:29.225617+0000","last_active":"2024-04-24T02:00:30.273895+0000","last_peered":"2024-04-24T02:00:30.273895+0000","last_clean":"2024-04-24T02:00:30.273895+0000","last_became_active":"2024-04-24T01:55:30.321866+0000","last_became_peered":"2024-04-24T01:55:30.321866+0000","last_unstale":"2024-04-24T02:00:30.273895+0000","last_undegraded":"2024-04-24T02:00:30.273895+0000","last_fullsized":"2024-04-24T02:00:30.273895+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":358,"log_dups_size":0,"ondisk_log_size":358,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":33,"snaptrim_duration":0,"stat_sum":{"num_bytes":2269077,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":201,"num_read_kb":136088,"num_write":180,"num_write_kb":48071,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"264'229","reported_seq":1148,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-04-24T02:00:32.209122+0000","last_change":"2024-04-24T02:00:32.219091+0000","last_active":"2024-04-24T02:00:32.209122+0000","last_peered":"2024-04-24T02:00:32.209122+0000","last_clean":"2024-04-24T02:00:32.209122+0000","last_became_active":"2024-04-24T01:55:30.320186+0000","last_became_peered":"2024-04-24T01:55:30.320186+0000","last_unstale":"2024-04-24T02:00:32.209122+0000","last_undegraded":"2024-04-24T02:00:32.209122+0000","last_fullsized":"2024-04-24T02:00:32.209122+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":229,"log_dups_size":0,"ondisk_log_size":229,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":24,"snaptrim_duration":0,"stat_sum":{"num_bytes":5159914,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":137,"num_read_kb":94252,"num_write":124,"num_write_kb":34018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"315'294","reported_seq":1220,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-04-24T02:00:33.209463+0000","last_change":"2024-04-24T02:00:32.239896+0000","last_active":"2024-04-24T02:00:33.209463+0000","last_peered":"2024-04-24T02:00:33.209463+0000","last_clean":"2024-04-24T02:00:33.209463+0000","last_became_active":"2024-04-24T01:55:30.288081+0000","last_became_peered":"2024-04-24T01:55:30.288081+0000","last_unstale":"2024-04-24T02:00:33.209463+0000","last_undegraded":"2024-04-24T02:00:33.209463+0000","last_fullsized":"2024-04-24T02:00:33.209463+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":294,"log_dups_size":0,"ondisk_log_size":294,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":30,"snaptrim_duration":0,"stat_sum":{"num_bytes":3395113,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":150,"num_read_kb":109442,"num_write":140,"num_write_kb":37310,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"314'125","reported_seq":1064,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-04-24T02:00:32.316615+0000","last_change":"2024-04-24T02:00:31.253786+0000","last_active":"2024-04-24T02:00:32.316615+0000","last_peered":"2024-04-24T02:00:32.316615+0000","last_clean":"2024-04-24T02:00:32.316615+0000","last_became_active":"2024-04-24T01:55:30.333815+0000","last_became_peered":"2024-04-24T01:55:30.333815+0000","last_unstale":"2024-04-24T02:00:32.316615+0000","last_undegraded":"2024-04-24T02:00:32.316615+0000","last_fullsized":"2024-04-24T02:00:32.316615+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":125,"log_dups_size":0,"ondisk_log_size":125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":15,"snaptrim_duration":0,"stat_sum":{"num_bytes":2588410,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":69,"num_read_kb":40384,"num_write":71,"num_write_kb":18850,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"315'611","reported_seq":1479,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-04-24T02:00:33.210126+0000","last_change":"2024-04-24T02:00:32.270660+0000","last_active":"2024-04-24T02:00:33.210126+0000","last_peered":"2024-04-24T02:00:33.210126+0000","last_clean":"2024-04-24T02:00:33.210126+0000","last_became_active":"2024-04-24T01:55:30.315075+0000","last_became_peered":"2024-04-24T01:55:30.315075+0000","last_unstale":"2024-04-24T02:00:33.210126+0000","last_undegraded":"2024-04-24T02:00:33.210126+0000","last_fullsized":"2024-04-24T02:00:33.210126+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":611,"log_dups_size":0,"ondisk_log_size":611,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":64,"snaptrim_duration":0,"stat_sum":{"num_bytes":7580353,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":272,"num_read_kb":173939,"num_write":272,"num_write_kb":70305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"315'117","reported_seq":1068,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-04-24T02:00:33.215967+0000","last_change":"2024-04-24T02:00:32.245177+0000","last_active":"2024-04-24T02:00:33.215967+0000","last_peered":"2024-04-24T02:00:33.215967+0000","last_clean":"2024-04-24T02:00:33.215967+0000","last_became_active":"2024-04-24T01:55:30.350300+0000","last_became_peered":"2024-04-24T01:55:30.350300+0000","last_unstale":"2024-04-24T02:00:33.215967+0000","last_undegraded":"2024-04-24T02:00:33.215967+0000","last_fullsized":"2024-04-24T02:00:33.215967+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":10,"snaptrim_duration":0,"stat_sum":{"num_bytes":6689384,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":63,"num_read_kb":40458,"num_write":77,"num_write_kb":19593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"264'34","reported_seq":958,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-04-24T02:00:32.225136+0000","last_change":"2024-04-24T02:00:32.234433+0000","last_active":"2024-04-24T02:00:32.225136+0000","last_peered":"2024-04-24T02:00:32.225136+0000","last_clean":"2024-04-24T02:00:32.225136+0000","last_became_active":"2024-04-24T01:55:30.376813+0000","last_became_peered":"2024-04-24T01:55:30.376813+0000","last_unstale":"2024-04-24T02:00:32.225136+0000","last_undegraded":"2024-04-24T02:00:32.225136+0000","last_fullsized":"2024-04-24T02:00:32.225136+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0,"stat_sum":{"num_bytes":3025359,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":10082,"num_write":27,"num_write_kb":6802,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'97","reported_seq":939,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-04-24T02:00:32.301278+0000","last_change":"2024-04-24T01:55:16.196470+0000","last_active":"2024-04-24T02:00:32.301278+0000","last_peered":"2024-04-24T02:00:32.301278+0000","last_clean":"2024-04-24T02:00:32.301278+0000","last_became_active":"2024-04-24T01:55:16.190464+0000","last_became_peered":"2024-04-24T01:55:16.190464+0000","last_unstale":"2024-04-24T02:00:32.301278+0000","last_undegraded":"2024-04-24T02:00:32.301278+0000","last_fullsized":"2024-04-24T02:00:32.301278+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:15.127115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:15.127115+0000","last_clean_scrub_stamp":"2024-04-24T01:55:15.127115+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":788,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-04-24T02:00:32.226921+0000","last_change":"2024-04-24T01:55:18.236148+0000","last_active":"2024-04-24T02:00:32.226921+0000","last_peered":"2024-04-24T02:00:32.226921+0000","last_clean":"2024-04-24T02:00:32.226921+0000","last_became_active":"2024-04-24T01:55:18.230154+0000","last_became_peered":"2024-04-24T01:55:18.230154+0000","last_unstale":"2024-04-24T02:00:32.226921+0000","last_undegraded":"2024-04-24T02:00:32.226921+0000","last_fullsized":"2024-04-24T02:00:32.226921+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"306'439","reported_seq":1346,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-04-24T02:00:32.224012+0000","last_change":"2024-04-24T02:00:32.231791+0000","last_active":"2024-04-24T02:00:32.224012+0000","last_peered":"2024-04-24T02:00:32.224012+0000","last_clean":"2024-04-24T02:00:32.224012+0000","last_became_active":"2024-04-24T01:55:30.316222+0000","last_became_peered":"2024-04-24T01:55:30.316222+0000","last_unstale":"2024-04-24T02:00:32.224012+0000","last_undegraded":"2024-04-24T02:00:32.224012+0000","last_fullsized":"2024-04-24T02:00:32.224012+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":439,"log_dups_size":0,"ondisk_log_size":439,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":38,"snaptrim_duration":0,"stat_sum":{"num_bytes":6202014,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":230,"num_read_kb":163232,"num_write":230,"num_write_kb":64433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":813,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-04-24T02:00:33.218083+0000","last_change":"2024-04-24T01:55:18.265905+0000","last_active":"2024-04-24T02:00:33.218083+0000","last_peered":"2024-04-24T02:00:33.218083+0000","last_clean":"2024-04-24T02:00:33.218083+0000","last_became_active":"2024-04-24T01:55:18.257020+0000","last_became_peered":"2024-04-24T01:55:18.257020+0000","last_unstale":"2024-04-24T02:00:33.218083+0000","last_undegraded":"2024-04-24T02:00:33.218083+0000","last_fullsized":"2024-04-24T02:00:33.218083+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"315'343","reported_seq":1245,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-04-24T02:00:32.375910+0000","last_change":"2024-04-24T02:00:31.216002+0000","last_active":"2024-04-24T02:00:32.375910+0000","last_peered":"2024-04-24T02:00:32.375910+0000","last_clean":"2024-04-24T02:00:32.375910+0000","last_became_active":"2024-04-24T01:55:30.344129+0000","last_became_peered":"2024-04-24T01:55:30.344129+0000","last_unstale":"2024-04-24T02:00:32.375910+0000","last_undegraded":"2024-04-24T02:00:32.375910+0000","last_fullsized":"2024-04-24T02:00:32.375910+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":343,"log_dups_size":0,"ondisk_log_size":343,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":29,"snaptrim_duration":0,"stat_sum":{"num_bytes":8655968,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":145,"num_read_kb":82337,"num_write":167,"num_write_kb":47096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":788,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-04-24T02:00:32.212320+0000","last_change":"2024-04-24T01:55:18.215251+0000","last_active":"2024-04-24T02:00:32.212320+0000","last_peered":"2024-04-24T02:00:32.212320+0000","last_clean":"2024-04-24T02:00:32.212320+0000","last_became_active":"2024-04-24T01:55:18.206605+0000","last_became_peered":"2024-04-24T01:55:18.206605+0000","last_unstale":"2024-04-24T02:00:32.212320+0000","last_undegraded":"2024-04-24T02:00:32.212320+0000","last_fullsized":"2024-04-24T02:00:32.212320+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"314'190","reported_seq":1138,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-04-24T02:00:32.311378+0000","last_change":"2024-04-24T02:00:31.223294+0000","last_active":"2024-04-24T02:00:32.311378+0000","last_peered":"2024-04-24T02:00:32.311378+0000","last_clean":"2024-04-24T02:00:32.311378+0000","last_became_active":"2024-04-24T01:55:30.401366+0000","last_became_peered":"2024-04-24T01:55:30.401366+0000","last_unstale":"2024-04-24T02:00:32.311378+0000","last_undegraded":"2024-04-24T02:00:32.311378+0000","last_fullsized":"2024-04-24T02:00:32.311378+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":190,"log_dups_size":0,"ondisk_log_size":190,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":16,"snaptrim_duration":0,"stat_sum":{"num_bytes":2437147,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":144,"num_read_kb":111400,"num_write":79,"num_write_kb":19750,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":813,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-04-24T02:00:33.217256+0000","last_change":"2024-04-24T01:55:18.230893+0000","last_active":"2024-04-24T02:00:33.217256+0000","last_peered":"2024-04-24T02:00:33.217256+0000","last_clean":"2024-04-24T02:00:33.217256+0000","last_became_active":"2024-04-24T01:55:18.225188+0000","last_became_peered":"2024-04-24T01:55:18.225188+0000","last_unstale":"2024-04-24T02:00:33.217256+0000","last_undegraded":"2024-04-24T02:00:33.217256+0000","last_fullsized":"2024-04-24T02:00:33.217256+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"302'275","reported_seq":1170,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-04-24T02:00:30.278299+0000","last_change":"2024-04-24T02:00:29.224473+0000","last_active":"2024-04-24T02:00:30.278299+0000","last_peered":"2024-04-24T02:00:30.278299+0000","last_clean":"2024-04-24T02:00:30.278299+0000","last_became_active":"2024-04-24T01:55:30.343318+0000","last_became_peered":"2024-04-24T01:55:30.343318+0000","last_unstale":"2024-04-24T02:00:30.278299+0000","last_undegraded":"2024-04-24T02:00:30.278299+0000","last_fullsized":"2024-04-24T02:00:30.278299+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":275,"log_dups_size":0,"ondisk_log_size":275,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":25,"snaptrim_duration":0,"stat_sum":{"num_bytes":10067723,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":150,"num_read_kb":122484,"num_write":129,"num_write_kb":35948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"315'413","reported_seq":1307,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-04-24T02:00:32.249332+0000","last_change":"2024-04-24T02:00:32.273405+0000","last_active":"2024-04-24T02:00:32.249332+0000","last_peered":"2024-04-24T02:00:32.249332+0000","last_clean":"2024-04-24T02:00:32.249332+0000","last_became_active":"2024-04-24T01:55:30.345308+0000","last_became_peered":"2024-04-24T01:55:30.345308+0000","last_unstale":"2024-04-24T02:00:32.249332+0000","last_undegraded":"2024-04-24T02:00:32.249332+0000","last_fullsized":"2024-04-24T02:00:32.249332+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":413,"log_dups_size":0,"ondisk_log_size":413,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":36,"snaptrim_duration":0,"stat_sum":{"num_bytes":9866039,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":210,"num_read_kb":136585,"num_write":201,"num_write_kb":53738,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":778,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-04-24T02:00:30.283508+0000","last_change":"2024-04-24T01:55:18.224238+0000","last_active":"2024-04-24T02:00:30.283508+0000","last_peered":"2024-04-24T02:00:30.283508+0000","last_clean":"2024-04-24T02:00:30.283508+0000","last_became_active":"2024-04-24T01:55:18.219131+0000","last_became_peered":"2024-04-24T01:55:18.219131+0000","last_unstale":"2024-04-24T02:00:30.283508+0000","last_undegraded":"2024-04-24T02:00:30.283508+0000","last_fullsized":"2024-04-24T02:00:30.283508+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"306'83","reported_seq":995,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-04-24T02:00:30.276459+0000","last_change":"2024-04-24T02:00:29.240068+0000","last_active":"2024-04-24T02:00:30.276459+0000","last_peered":"2024-04-24T02:00:30.276459+0000","last_clean":"2024-04-24T02:00:30.276459+0000","last_became_active":"2024-04-24T01:55:30.319958+0000","last_became_peered":"2024-04-24T01:55:30.319958+0000","last_unstale":"2024-04-24T02:00:30.276459+0000","last_undegraded":"2024-04-24T02:00:30.276459+0000","last_fullsized":"2024-04-24T02:00:30.276459+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":83,"log_dups_size":0,"ondisk_log_size":83,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":7,"snaptrim_duration":0,"stat_sum":{"num_bytes":3882516,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":41,"num_read_kb":36474,"num_write":51,"num_write_kb":14818,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":806,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-04-24T02:00:32.312823+0000","last_change":"2024-04-24T01:55:18.235763+0000","last_active":"2024-04-24T02:00:32.312823+0000","last_peered":"2024-04-24T02:00:32.312823+0000","last_clean":"2024-04-24T02:00:32.312823+0000","last_became_active":"2024-04-24T01:55:18.229175+0000","last_became_peered":"2024-04-24T01:55:18.229175+0000","last_unstale":"2024-04-24T02:00:32.312823+0000","last_undegraded":"2024-04-24T02:00:32.312823+0000","last_fullsized":"2024-04-24T02:00:32.312823+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"315'170","reported_seq":1097,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-04-24T02:00:32.357931+0000","last_change":"2024-04-24T02:00:31.216811+0000","last_active":"2024-04-24T02:00:32.357931+0000","last_peered":"2024-04-24T02:00:32.357931+0000","last_clean":"2024-04-24T02:00:32.357931+0000","last_became_active":"2024-04-24T01:55:30.372690+0000","last_became_peered":"2024-04-24T01:55:30.372690+0000","last_unstale":"2024-04-24T02:00:32.357931+0000","last_undegraded":"2024-04-24T02:00:32.357931+0000","last_fullsized":"2024-04-24T02:00:32.357931+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":15,"snaptrim_duration":0,"stat_sum":{"num_bytes":5227241,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":81,"num_read_kb":56406,"num_write":85,"num_write_kb":24562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":778,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-04-24T02:00:30.283473+0000","last_change":"2024-04-24T01:55:18.278351+0000","last_active":"2024-04-24T02:00:30.283473+0000","last_peered":"2024-04-24T02:00:30.283473+0000","last_clean":"2024-04-24T02:00:30.283473+0000","last_became_active":"2024-04-24T01:55:18.273668+0000","last_became_peered":"2024-04-24T01:55:18.273668+0000","last_unstale":"2024-04-24T02:00:30.283473+0000","last_undegraded":"2024-04-24T02:00:30.283473+0000","last_fullsized":"2024-04-24T02:00:30.283473+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"306'349","reported_seq":1264,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-04-24T02:00:30.285993+0000","last_change":"2024-04-24T02:00:29.240791+0000","last_active":"2024-04-24T02:00:30.285993+0000","last_peered":"2024-04-24T02:00:30.285993+0000","last_clean":"2024-04-24T02:00:30.285993+0000","last_became_active":"2024-04-24T01:55:30.363123+0000","last_became_peered":"2024-04-24T01:55:30.363123+0000","last_unstale":"2024-04-24T02:00:30.285993+0000","last_undegraded":"2024-04-24T02:00:30.285993+0000","last_fullsized":"2024-04-24T02:00:30.285993+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":349,"log_dups_size":0,"ondisk_log_size":349,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":41,"snaptrim_duration":0,"stat_sum":{"num_bytes":7428892,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":227,"num_read_kb":196496,"num_write":180,"num_write_kb":48260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":788,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-04-24T02:00:32.226712+0000","last_change":"2024-04-24T01:55:18.241117+0000","last_active":"2024-04-24T02:00:32.226712+0000","last_peered":"2024-04-24T02:00:32.226712+0000","last_clean":"2024-04-24T02:00:32.226712+0000","last_became_active":"2024-04-24T01:55:18.234313+0000","last_became_peered":"2024-04-24T01:55:18.234313+0000","last_unstale":"2024-04-24T02:00:32.226712+0000","last_undegraded":"2024-04-24T02:00:32.226712+0000","last_fullsized":"2024-04-24T02:00:32.226712+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":88845234,"num_objects":37,"num_object_clones":5,"num_object_copies":111,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":2307,"num_read_kb":1662904,"num_write":2124,"num_write_kb":575665,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":174858240,"data_stored":179709858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4272,"ondisk_log_size":4272,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":1}],"osd_stats":[{"osd":3,"up_from":10,"seq":42949673279,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":84200,"kb_used_data":48832,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94287640,"statfs":{"total":96636764160,"available":96550543360,"internally_reserved":0,"allocated":50003968,"data_stored":50654939,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673279,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":74424,"kb_used_data":39280,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94297416,"statfs":{"total":96636764160,"available":96560553984,"internally_reserved":0,"allocated":40222720,"data_stored":41009401,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705984,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":74560,"kb_used_data":39200,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94297280,"statfs":{"total":96636764160,"available":96560414720,"internally_reserved":0,"allocated":40140800,"data_stored":41755997,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705983,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":82908,"kb_used_data":47660,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94288932,"statfs":{"total":96636764160,"available":96551866368,"internally_reserved":0,"allocated":48803840,"data_stored":49895035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":39059456,"data_stored":40712686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":47669248,"data_stored":49390982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38944768,"data_stored":39877962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49184768,"data_stored":49728228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T02:00:35.338 INFO:tasks.rados.rados.0.smithi079.stdout:1314: finishing write tid 3 to smithi07929762-39 2024-04-24T02:00:35.338 INFO:tasks.rados.rados.0.smithi079.stdout:1319: finishing write tid 2 to smithi07929762-36 2024-04-24T02:00:35.338 INFO:tasks.rados.rados.0.smithi079.stdout:1316: finishing write tid 1 to smithi07929762-10 2024-04-24T02:00:35.338 INFO:tasks.rados.rados.0.smithi079.stdout:1319: finishing write tid 3 to smithi07929762-36 2024-04-24T02:00:35.338 INFO:tasks.rados.rados.0.smithi079.stdout:1319: finishing write tid 4 to smithi07929762-36 2024-04-24T02:00:35.338 INFO:tasks.rados.rados.0.smithi079.stdout:1319: oid 36 updating version 0 to 128 2024-04-24T02:00:35.338 INFO:tasks.rados.rados.0.smithi079.stdout:1319: oid 36 version 128 is already newer than 128 2024-04-24T02:00:35.338 INFO:tasks.rados.rados.0.smithi079.stdout:1319: oid 36 version 128 is already newer than 128 2024-04-24T02:00:35.338 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 36 v 128 (ObjNum 646 snap 151 seq_num 646) dirty exists 2024-04-24T02:00:35.338 INFO:tasks.rados.rados.0.smithi079.stdout:1319: left oid 36 (ObjNum 646 snap 151 seq_num 646) 2024-04-24T02:00:35.339 INFO:tasks.rados.rados.0.smithi079.stdout:1314: finishing write tid 4 to smithi07929762-39 2024-04-24T02:00:35.339 INFO:tasks.rados.rados.0.smithi079.stdout:1314: oid 39 updating version 0 to 279 2024-04-24T02:00:35.339 INFO:tasks.rados.rados.0.smithi079.stdout:1314: oid 39 version 279 is already newer than 279 2024-04-24T02:00:35.339 INFO:tasks.rados.rados.0.smithi079.stdout:1314: oid 39 version 279 is already newer than 279 2024-04-24T02:00:35.339 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 39 v 279 (ObjNum 642 snap 151 seq_num 642) dirty exists 2024-04-24T02:00:35.339 INFO:tasks.rados.rados.0.smithi079.stdout:1314: left oid 39 (ObjNum 642 snap 151 seq_num 642) 2024-04-24T02:00:35.339 INFO:tasks.rados.rados.0.smithi079.stdout:1316: finishing write tid 2 to smithi07929762-10 2024-04-24T02:00:35.339 INFO:tasks.rados.rados.0.smithi079.stdout:1316: finishing write tid 3 to smithi07929762-10 2024-04-24T02:00:35.339 INFO:tasks.rados.rados.0.smithi079.stdout:1316: finishing write tid 4 to smithi07929762-10 2024-04-24T02:00:35.339 INFO:tasks.rados.rados.0.smithi079.stdout:1316: finishing write tid 5 to smithi07929762-10 2024-04-24T02:00:35.339 INFO:tasks.rados.rados.0.smithi079.stdout:1316: oid 10 updating version 0 to 365 2024-04-24T02:00:35.339 INFO:tasks.rados.rados.0.smithi079.stdout:1316: oid 10 version 365 is already newer than 362 2024-04-24T02:00:35.339 INFO:tasks.rados.rados.0.smithi079.stdout:1316: oid 10 version 365 is already newer than 365 2024-04-24T02:00:35.339 INFO:tasks.rados.rados.0.smithi079.stdout:1316: oid 10 version 365 is already newer than 365 2024-04-24T02:00:35.339 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 10 v 365 (ObjNum 644 snap 151 seq_num 644) dirty exists 2024-04-24T02:00:35.339 INFO:tasks.rados.rados.0.smithi079.stdout:1316: left oid 10 (ObjNum 644 snap 151 seq_num 644) 2024-04-24T02:00:35.604 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T02:00:35.709 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:35.708+0000 7fc0df606640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:35.713 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:35.713+0000 7fc0df606640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:36.197 INFO:tasks.rados.rados.0.smithi079.stdout:1314: done (6 left) 2024-04-24T02:00:36.198 INFO:tasks.rados.rados.0.smithi079.stdout:1316: done (5 left) 2024-04-24T02:00:36.198 INFO:tasks.rados.rados.0.smithi079.stdout:1317: done (4 left) 2024-04-24T02:00:36.198 INFO:tasks.rados.rados.0.smithi079.stdout:1318: done (3 left) 2024-04-24T02:00:36.198 INFO:tasks.rados.rados.0.smithi079.stdout:1319: done (2 left) 2024-04-24T02:00:36.198 INFO:tasks.rados.rados.0.smithi079.stdout:1320: done (1 left) 2024-04-24T02:00:36.198 INFO:tasks.rados.rados.0.smithi079.stdout:1322: done (0 left) 2024-04-24T02:00:36.198 INFO:tasks.rados.rados.0.smithi079.stdout:1323: read oid 22 snap -1 2024-04-24T02:00:36.198 INFO:tasks.rados.rados.0.smithi079.stdout:1323: expect (ObjNum 621 snap 143 seq_num 621) 2024-04-24T02:00:36.198 INFO:tasks.rados.rados.0.smithi079.stdout:1324: read oid 50 snap 151 2024-04-24T02:00:36.198 INFO:tasks.rados.rados.0.smithi079.stdout:1324: expect deleted 2024-04-24T02:00:36.198 INFO:tasks.rados.rados.0.smithi079.stdout:1325: write oid 30 current snap is 152 2024-04-24T02:00:36.198 INFO:tasks.rados.rados.0.smithi079.stdout:1325: seq_num 647 ranges {474129=576108,1627732=587214,2718088=156041} 2024-04-24T02:00:36.203 INFO:tasks.rados.rados.0.smithi079.stdout:1325: writing smithi07929762-30 from 474129 to 1050237 tid 1 2024-04-24T02:00:36.209 INFO:tasks.rados.rados.0.smithi079.stdout:1325: writing smithi07929762-30 from 1627732 to 2214946 tid 2 2024-04-24T02:00:36.214 INFO:tasks.rados.rados.0.smithi079.stdout:1325: writing smithi07929762-30 from 2718088 to 2874129 tid 3 2024-04-24T02:00:36.214 INFO:tasks.rados.rados.0.smithi079.stdout:1326: snap_remove snap 148 2024-04-24T02:00:37.213 INFO:tasks.rados.rados.0.smithi079.stdout:1326: done (3 left) 2024-04-24T02:00:37.213 INFO:tasks.rados.rados.0.smithi079.stdout:1327: read oid 43 snap -1 2024-04-24T02:00:37.213 INFO:tasks.rados.rados.0.smithi079.stdout:1327: expect (ObjNum 623 snap 144 seq_num 623) 2024-04-24T02:00:37.213 INFO:tasks.rados.rados.0.smithi079.stdout:1328: snap_create 2024-04-24T02:00:37.213 INFO:tasks.rados.rados.0.smithi079.stdout:1323: expect (ObjNum 621 snap 143 seq_num 621) 2024-04-24T02:00:37.369 INFO:tasks.rados.rados.0.smithi079.stdout:1325: finishing write tid 1 to smithi07929762-30 2024-04-24T02:00:37.370 INFO:tasks.rados.rados.0.smithi079.stdout:1325: finishing write tid 2 to smithi07929762-30 2024-04-24T02:00:37.370 INFO:tasks.rados.rados.0.smithi079.stdout:1325: finishing write tid 3 to smithi07929762-30 2024-04-24T02:00:37.370 INFO:tasks.rados.rados.0.smithi079.stdout:1325: finishing write tid 4 to smithi07929762-30 2024-04-24T02:00:37.370 INFO:tasks.rados.rados.0.smithi079.stdout:1325: finishing write tid 5 to smithi07929762-30 2024-04-24T02:00:37.370 INFO:tasks.rados.rados.0.smithi079.stdout:1325: oid 30 updating version 0 to 177 2024-04-24T02:00:37.370 INFO:tasks.rados.rados.0.smithi079.stdout:1325: oid 30 version 177 is already newer than 177 2024-04-24T02:00:37.370 INFO:tasks.rados.rados.0.smithi079.stdout:1325: oid 30 version 177 is already newer than 177 2024-04-24T02:00:37.370 INFO:tasks.rados.rados.0.smithi079.stdout:1325: oid 30 version 177 is already newer than 177 2024-04-24T02:00:37.370 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 30 v 177 (ObjNum 647 snap 152 seq_num 647) dirty exists 2024-04-24T02:00:37.370 INFO:tasks.rados.rados.0.smithi079.stdout:1325: left oid 30 (ObjNum 647 snap 152 seq_num 647) 2024-04-24T02:00:37.370 INFO:tasks.rados.rados.0.smithi079.stdout:1327: expect (ObjNum 623 snap 144 seq_num 623) 2024-04-24T02:00:37.804 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T02:00:37.910 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:37.909+0000 7f4f4af5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:37.913 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:37.914+0000 7f4f4af5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:38.211 INFO:tasks.rados.rados.0.smithi079.stdout:1323: done (4 left) 2024-04-24T02:00:38.211 INFO:tasks.rados.rados.0.smithi079.stdout:1324: done (3 left) 2024-04-24T02:00:38.211 INFO:tasks.rados.rados.0.smithi079.stdout:1325: done (2 left) 2024-04-24T02:00:38.211 INFO:tasks.rados.rados.0.smithi079.stdout:1327: done (1 left) 2024-04-24T02:00:38.211 INFO:tasks.rados.rados.0.smithi079.stdout:1328: done (0 left) 2024-04-24T02:00:38.211 INFO:tasks.rados.rados.0.smithi079.stdout:1329: read oid 45 snap -1 2024-04-24T02:00:38.211 INFO:tasks.rados.rados.0.smithi079.stdout:1329: expect deleted 2024-04-24T02:00:38.211 INFO:tasks.rados.rados.0.smithi079.stdout:1330: delete oid 15 current snap is 153 2024-04-24T02:00:38.291 INFO:tasks.rados.rados.0.smithi079.stdout:1329: done (1 left) 2024-04-24T02:00:38.292 INFO:tasks.rados.rados.0.smithi079.stdout:1330: done (0 left) 2024-04-24T02:00:38.292 INFO:tasks.rados.rados.0.smithi079.stdout:1331: delete oid 30 current snap is 153 2024-04-24T02:00:38.356 INFO:tasks.rados.rados.0.smithi079.stdout:1331: done (0 left) 2024-04-24T02:00:38.356 INFO:tasks.rados.rados.0.smithi079.stdout:1332: read oid 36 snap -1 2024-04-24T02:00:38.356 INFO:tasks.rados.rados.0.smithi079.stdout:1332: expect (ObjNum 646 snap 151 seq_num 646) 2024-04-24T02:00:38.356 INFO:tasks.rados.rados.0.smithi079.stdout:1333: read oid 9 snap 151 2024-04-24T02:00:38.356 INFO:tasks.rados.rados.0.smithi079.stdout:1333: expect (ObjNum 577 snap 132 seq_num 577) 2024-04-24T02:00:38.356 INFO:tasks.rados.rados.0.smithi079.stdout:1334: read oid 43 snap 149 2024-04-24T02:00:38.356 INFO:tasks.rados.rados.0.smithi079.stdout:1334: expect (ObjNum 623 snap 144 seq_num 623) 2024-04-24T02:00:38.356 INFO:tasks.rados.rados.0.smithi079.stdout:1335: read oid 15 snap -1 2024-04-24T02:00:38.356 INFO:tasks.rados.rados.0.smithi079.stdout:1335: expect deleted 2024-04-24T02:00:38.356 INFO:tasks.rados.rados.0.smithi079.stdout:1336: delete oid 27 current snap is 153 2024-04-24T02:00:38.386 INFO:tasks.rados.rados.0.smithi079.stdout:1333: expect (ObjNum 577 snap 132 seq_num 577) 2024-04-24T02:00:38.622 INFO:tasks.rados.rados.0.smithi079.stdout:1332: expect (ObjNum 646 snap 151 seq_num 646) 2024-04-24T02:00:38.729 INFO:tasks.rados.rados.0.smithi079.stdout:1334: expect (ObjNum 623 snap 144 seq_num 623) 2024-04-24T02:00:38.880 INFO:tasks.rados.rados.0.smithi079.stdout:1332: done (4 left) 2024-04-24T02:00:38.880 INFO:tasks.rados.rados.0.smithi079.stdout:1333: done (3 left) 2024-04-24T02:00:38.880 INFO:tasks.rados.rados.0.smithi079.stdout:1334: done (2 left) 2024-04-24T02:00:38.880 INFO:tasks.rados.rados.0.smithi079.stdout:1335: done (1 left) 2024-04-24T02:00:38.880 INFO:tasks.rados.rados.0.smithi079.stdout:1336: done (0 left) 2024-04-24T02:00:38.880 INFO:tasks.rados.rados.0.smithi079.stdout:1337: delete oid 49 current snap is 153 2024-04-24T02:00:38.888 INFO:tasks.rados.rados.0.smithi079.stdout:1337: done (0 left) 2024-04-24T02:00:38.888 INFO:tasks.rados.rados.0.smithi079.stdout:1338: read oid 48 snap -1 2024-04-24T02:00:38.888 INFO:tasks.rados.rados.0.smithi079.stdout:1338: expect (ObjNum 609 snap 142 seq_num 609) 2024-04-24T02:00:38.888 INFO:tasks.rados.rados.0.smithi079.stdout:1339: write (excl) oid 2 current snap is 153 2024-04-24T02:00:38.888 INFO:tasks.rados.rados.0.smithi079.stdout:1339: seq_num 652 ranges {558164=552353,1803789=454563,2855203=102961} 2024-04-24T02:00:38.897 INFO:tasks.rados.rados.0.smithi079.stdout:1339: writing smithi07929762-2 from 558164 to 1110517 tid 1 2024-04-24T02:00:38.905 INFO:tasks.rados.rados.0.smithi079.stdout:1339: writing smithi07929762-2 from 1803789 to 2258352 tid 2 2024-04-24T02:00:38.910 INFO:tasks.rados.rados.0.smithi079.stdout:1339: writing smithi07929762-2 from 2855203 to 2958164 tid 3 2024-04-24T02:00:38.910 INFO:tasks.rados.rados.0.smithi079.stdout:1340: snap_create 2024-04-24T02:00:38.932 INFO:tasks.rados.rados.0.smithi079.stdout:1338: expect (ObjNum 609 snap 142 seq_num 609) 2024-04-24T02:00:39.076 INFO:tasks.rados.rados.0.smithi079.stdout:1339: finishing write tid 1 to smithi07929762-2 2024-04-24T02:00:39.076 INFO:tasks.rados.rados.0.smithi079.stdout:1339: finishing write tid 2 to smithi07929762-2 2024-04-24T02:00:39.076 INFO:tasks.rados.rados.0.smithi079.stdout:1339: finishing write tid 3 to smithi07929762-2 2024-04-24T02:00:39.076 INFO:tasks.rados.rados.0.smithi079.stdout:1339: finishing write tid 4 to smithi07929762-2 2024-04-24T02:00:39.076 INFO:tasks.rados.rados.0.smithi079.stdout:1339: finishing write tid 5 to smithi07929762-2 2024-04-24T02:00:39.076 INFO:tasks.rados.rados.0.smithi079.stdout:1339: oid 2 updating version 0 to 88 2024-04-24T02:00:39.076 INFO:tasks.rados.rados.0.smithi079.stdout:1339: oid 2 version 88 is already newer than 88 2024-04-24T02:00:39.076 INFO:tasks.rados.rados.0.smithi079.stdout:1339: oid 2 version 88 is already newer than 88 2024-04-24T02:00:39.076 INFO:tasks.rados.rados.0.smithi079.stdout:1339: oid 2 version 88 is already newer than 85 2024-04-24T02:00:39.076 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 2 v 88 (ObjNum 652 snap 153 seq_num 652) dirty exists 2024-04-24T02:00:39.077 INFO:tasks.rados.rados.0.smithi079.stdout:1339: left oid 2 (ObjNum 652 snap 153 seq_num 652) 2024-04-24T02:00:39.221 INFO:tasks.rados.rados.0.smithi079.stdout:1338: done (2 left) 2024-04-24T02:00:39.221 INFO:tasks.rados.rados.0.smithi079.stdout:1339: done (1 left) 2024-04-24T02:00:39.221 INFO:tasks.rados.rados.0.smithi079.stdout:1340: done (0 left) 2024-04-24T02:00:39.221 INFO:tasks.rados.rados.0.smithi079.stdout:1341: write oid 23 current snap is 154 2024-04-24T02:00:39.221 INFO:tasks.rados.rados.0.smithi079.stdout:1341: seq_num 653 ranges {574971=627602,1862271=598221} 2024-04-24T02:00:39.231 INFO:tasks.rados.rados.0.smithi079.stdout:1341: writing smithi07929762-23 from 574971 to 1202573 tid 1 2024-04-24T02:00:39.240 INFO:tasks.rados.rados.0.smithi079.stdout:1341: writing smithi07929762-23 from 1862271 to 2460492 tid 2 2024-04-24T02:00:39.241 INFO:tasks.rados.rados.0.smithi079.stdout:1342: snap_remove snap 152 2024-04-24T02:00:40.007 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T02:00:40.113 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:40.111+0000 7f30ba66d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:40.117 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:40.117+0000 7f30ba66d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:40.227 INFO:tasks.rados.rados.0.smithi079.stdout:1342: done (1 left) 2024-04-24T02:00:40.227 INFO:tasks.rados.rados.0.smithi079.stdout:1343: delete oid 21 current snap is 154 2024-04-24T02:00:40.227 INFO:tasks.rados.rados.0.smithi079.stdout:1341: finishing write tid 1 to smithi07929762-23 2024-04-24T02:00:40.227 INFO:tasks.rados.rados.0.smithi079.stdout:1341: finishing write tid 2 to smithi07929762-23 2024-04-24T02:00:40.227 INFO:tasks.rados.rados.0.smithi079.stdout:1341: finishing write tid 3 to smithi07929762-23 2024-04-24T02:00:40.228 INFO:tasks.rados.rados.0.smithi079.stdout:1341: finishing write tid 4 to smithi07929762-23 2024-04-24T02:00:40.228 INFO:tasks.rados.rados.0.smithi079.stdout:1341: oid 23 updating version 0 to 125 2024-04-24T02:00:40.228 INFO:tasks.rados.rados.0.smithi079.stdout:1341: oid 23 version 125 is already newer than 125 2024-04-24T02:00:40.228 INFO:tasks.rados.rados.0.smithi079.stdout:1341: oid 23 version 125 is already newer than 125 2024-04-24T02:00:40.228 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 23 v 125 (ObjNum 653 snap 154 seq_num 653) dirty exists 2024-04-24T02:00:40.228 INFO:tasks.rados.rados.0.smithi079.stdout:1341: left oid 23 (ObjNum 653 snap 154 seq_num 653) 2024-04-24T02:00:40.234 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-04-24T02:00:40.234 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-04-24T02:00:40.234 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-04-24T02:00:40.235 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-04-24T02:00:40.297 INFO:tasks.rados.rados.0.smithi079.stdout:1341: done (1 left) 2024-04-24T02:00:40.297 INFO:tasks.rados.rados.0.smithi079.stdout:1343: done (0 left) 2024-04-24T02:00:40.297 INFO:tasks.rados.rados.0.smithi079.stdout:1344: write oid 46 current snap is 154 2024-04-24T02:00:40.297 INFO:tasks.rados.rados.0.smithi079.stdout:1344: seq_num 655 ranges {608585=494453,1579235=601890,2821119=187466} 2024-04-24T02:00:40.304 INFO:tasks.rados.rados.0.smithi079.stdout:1344: writing smithi07929762-46 from 608585 to 1103038 tid 1 2024-04-24T02:00:40.310 INFO:tasks.rados.rados.0.smithi079.stdout:1344: writing smithi07929762-46 from 1579235 to 2181125 tid 2 2024-04-24T02:00:40.315 INFO:tasks.rados.rados.0.smithi079.stdout:1344: writing smithi07929762-46 from 2821119 to 3008585 tid 3 2024-04-24T02:00:40.315 INFO:tasks.rados.rados.0.smithi079.stdout:1345: write oid 25 current snap is 154 2024-04-24T02:00:40.316 INFO:tasks.rados.rados.0.smithi079.stdout:1345: seq_num 656 ranges {625392=569702,1637717=745548,2848542=176850} 2024-04-24T02:00:40.323 INFO:tasks.rados.rados.0.smithi079.stdout:1345: writing smithi07929762-25 from 625392 to 1195094 tid 1 2024-04-24T02:00:40.331 INFO:tasks.rados.rados.0.smithi079.stdout:1345: writing smithi07929762-25 from 1637717 to 2383265 tid 2 2024-04-24T02:00:40.334 INFO:tasks.rados.rados.0.smithi079.stdout:1345: writing smithi07929762-25 from 2848542 to 3025392 tid 3 2024-04-24T02:00:40.335 INFO:tasks.rados.rados.0.smithi079.stdout:1346: write (excl) oid 15 current snap is 154 2024-04-24T02:00:40.335 INFO:tasks.rados.rados.0.smithi079.stdout:1346: seq_num 657 ranges {642199=644951,1696199=605559,2992318=49881} 2024-04-24T02:00:40.344 INFO:tasks.rados.rados.0.smithi079.stdout:1346: writing smithi07929762-15 from 642199 to 1287150 tid 1 2024-04-24T02:00:40.351 INFO:tasks.rados.rados.0.smithi079.stdout:1346: writing smithi07929762-15 from 1696199 to 2301758 tid 2 2024-04-24T02:00:40.356 INFO:tasks.rados.rados.0.smithi079.stdout:1346: writing smithi07929762-15 from 2992318 to 3042199 tid 3 2024-04-24T02:00:40.356 INFO:tasks.rados.rados.0.smithi079.stdout:1347: snap_remove snap 147 2024-04-24T02:00:40.367 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:40.361+0000 7f58beaed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:40.373 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:40.373+0000 7f58beaed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:40.420 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T02:00:40.420 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T02:00:40.420 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T02:00:40.420 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T02:00:40.420 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T02:00:40.420 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T02:00:40.420 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T02:00:40.420 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T02:00:40.420 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T02:00:40.421 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T02:00:40.421 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T02:00:40.421 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T02:00:40.424 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T02:00:40.425 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T02:00:40.425 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-04-24T02:00:40.542 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:40.538+0000 7f41fcead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:40.546 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:40.546+0000 7f41fcead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:40.560 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T02:00:40.560 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T02:00:40.561 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T02:00:40.561 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T02:00:40.561 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T02:00:40.561 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T02:00:40.561 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T02:00:40.561 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T02:00:40.561 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T02:00:40.561 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T02:00:40.561 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T02:00:40.561 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T02:00:40.563 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T02:00:40.564 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T02:00:40.564 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-04-24T02:00:40.672 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:40.667+0000 7f9af639d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:40.675 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:40.675+0000 7f9af639d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:40.691 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T02:00:40.691 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T02:00:40.691 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T02:00:40.691 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T02:00:40.691 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T02:00:40.691 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T02:00:40.692 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T02:00:40.692 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T02:00:40.692 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T02:00:40.692 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T02:00:40.692 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T02:00:40.692 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T02:00:40.694 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T02:00:40.695 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T02:00:40.695 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-04-24T02:00:40.801 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:40.796+0000 7f25474ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:40.805 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:40.805+0000 7f25474ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:40.821 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T02:00:40.821 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T02:00:40.821 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T02:00:40.821 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T02:00:40.821 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T02:00:40.821 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T02:00:40.821 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T02:00:40.821 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T02:00:40.821 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T02:00:40.821 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T02:00:40.821 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T02:00:40.821 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T02:00:40.824 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T02:00:40.824 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T02:00:41.235 INFO:tasks.rados.rados.0.smithi079.stdout:1347: done (3 left) 2024-04-24T02:00:41.235 INFO:tasks.rados.rados.0.smithi079.stdout:1348: snap_remove snap 153 2024-04-24T02:00:42.211 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T02:00:42.237 INFO:tasks.rados.rados.0.smithi079.stdout:1348: done (3 left) 2024-04-24T02:00:42.237 INFO:tasks.rados.rados.0.smithi079.stdout:1349: snap_remove snap 149 2024-04-24T02:00:42.323 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:42.322+0000 7f8b90aad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:42.327 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:42.327+0000 7f8b90aad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:43.237 INFO:tasks.rados.rados.0.smithi079.stdout:1349: done (3 left) 2024-04-24T02:00:43.237 INFO:tasks.rados.rados.0.smithi079.stdout:1350: delete oid 24 current snap is 154 2024-04-24T02:00:43.237 INFO:tasks.rados.rados.0.smithi079.stdout:1346: finishing write tid 1 to smithi07929762-15 2024-04-24T02:00:43.237 INFO:tasks.rados.rados.0.smithi079.stdout:1344: finishing write tid 1 to smithi07929762-46 2024-04-24T02:00:43.237 INFO:tasks.rados.rados.0.smithi079.stdout:1344: finishing write tid 2 to smithi07929762-46 2024-04-24T02:00:43.237 INFO:tasks.rados.rados.0.smithi079.stdout:1344: finishing write tid 3 to smithi07929762-46 2024-04-24T02:00:43.238 INFO:tasks.rados.rados.0.smithi079.stdout:1346: finishing write tid 2 to smithi07929762-15 2024-04-24T02:00:43.238 INFO:tasks.rados.rados.0.smithi079.stdout:1344: finishing write tid 4 to smithi07929762-46 2024-04-24T02:00:43.238 INFO:tasks.rados.rados.0.smithi079.stdout:1344: finishing write tid 5 to smithi07929762-46 2024-04-24T02:00:43.238 INFO:tasks.rados.rados.0.smithi079.stdout:1344: oid 46 updating version 0 to 299 2024-04-24T02:00:43.238 INFO:tasks.rados.rados.0.smithi079.stdout:1344: oid 46 version 299 is already newer than 299 2024-04-24T02:00:43.238 INFO:tasks.rados.rados.0.smithi079.stdout:1344: oid 46 version 299 is already newer than 299 2024-04-24T02:00:43.238 INFO:tasks.rados.rados.0.smithi079.stdout:1344: oid 46 version 299 is already newer than 299 2024-04-24T02:00:43.238 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 46 v 299 (ObjNum 655 snap 154 seq_num 655) dirty exists 2024-04-24T02:00:43.238 INFO:tasks.rados.rados.0.smithi079.stdout:1344: left oid 46 (ObjNum 655 snap 154 seq_num 655) 2024-04-24T02:00:43.238 INFO:tasks.rados.rados.0.smithi079.stdout:1346: finishing write tid 3 to smithi07929762-15 2024-04-24T02:00:43.238 INFO:tasks.rados.rados.0.smithi079.stdout:1346: finishing write tid 4 to smithi07929762-15 2024-04-24T02:00:43.238 INFO:tasks.rados.rados.0.smithi079.stdout:1346: finishing write tid 5 to smithi07929762-15 2024-04-24T02:00:43.238 INFO:tasks.rados.rados.0.smithi079.stdout:1346: oid 15 updating version 0 to 351 2024-04-24T02:00:43.238 INFO:tasks.rados.rados.0.smithi079.stdout:1346: oid 15 updating version 351 to 354 2024-04-24T02:00:43.238 INFO:tasks.rados.rados.0.smithi079.stdout:1346: oid 15 version 354 is already newer than 354 2024-04-24T02:00:43.238 INFO:tasks.rados.rados.0.smithi079.stdout:1346: oid 15 version 354 is already newer than 354 2024-04-24T02:00:43.239 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 15 v 354 (ObjNum 657 snap 154 seq_num 657) dirty exists 2024-04-24T02:00:43.239 INFO:tasks.rados.rados.0.smithi079.stdout:1346: left oid 15 (ObjNum 657 snap 154 seq_num 657) 2024-04-24T02:00:43.239 INFO:tasks.rados.rados.0.smithi079.stdout:1345: finishing write tid 1 to smithi07929762-25 2024-04-24T02:00:43.239 INFO:tasks.rados.rados.0.smithi079.stdout:1345: finishing write tid 2 to smithi07929762-25 2024-04-24T02:00:43.239 INFO:tasks.rados.rados.0.smithi079.stdout:1345: finishing write tid 3 to smithi07929762-25 2024-04-24T02:00:43.239 INFO:tasks.rados.rados.0.smithi079.stdout:1345: finishing write tid 4 to smithi07929762-25 2024-04-24T02:00:43.239 INFO:tasks.rados.rados.0.smithi079.stdout:1345: finishing write tid 5 to smithi07929762-25 2024-04-24T02:00:43.239 INFO:tasks.rados.rados.0.smithi079.stdout:1345: oid 25 updating version 0 to 444 2024-04-24T02:00:43.239 INFO:tasks.rados.rados.0.smithi079.stdout:1345: oid 25 version 444 is already newer than 444 2024-04-24T02:00:43.239 INFO:tasks.rados.rados.0.smithi079.stdout:1345: oid 25 version 444 is already newer than 444 2024-04-24T02:00:43.239 INFO:tasks.rados.rados.0.smithi079.stdout:1345: oid 25 version 444 is already newer than 444 2024-04-24T02:00:43.239 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 25 v 444 (ObjNum 656 snap 154 seq_num 656) dirty exists 2024-04-24T02:00:43.239 INFO:tasks.rados.rados.0.smithi079.stdout:1345: left oid 25 (ObjNum 656 snap 154 seq_num 656) 2024-04-24T02:00:43.332 INFO:tasks.rados.rados.0.smithi079.stdout:1344: done (3 left) 2024-04-24T02:00:43.332 INFO:tasks.rados.rados.0.smithi079.stdout:1345: done (2 left) 2024-04-24T02:00:43.332 INFO:tasks.rados.rados.0.smithi079.stdout:1346: done (1 left) 2024-04-24T02:00:43.332 INFO:tasks.rados.rados.0.smithi079.stdout:1350: done (0 left) 2024-04-24T02:00:43.332 INFO:tasks.rados.rados.0.smithi079.stdout:1351: read oid 29 snap 151 2024-04-24T02:00:43.332 INFO:tasks.rados.rados.0.smithi079.stdout:1351: expect deleted 2024-04-24T02:00:43.332 INFO:tasks.rados.rados.0.smithi079.stdout:1352: read oid 27 snap -1 2024-04-24T02:00:43.332 INFO:tasks.rados.rados.0.smithi079.stdout:1352: expect deleted 2024-04-24T02:00:43.332 INFO:tasks.rados.rados.0.smithi079.stdout:1353: read oid 12 snap -1 2024-04-24T02:00:43.333 INFO:tasks.rados.rados.0.smithi079.stdout:1353: expect deleted 2024-04-24T02:00:43.333 INFO:tasks.rados.rados.0.smithi079.stdout:1354: read oid 8 snap -1 2024-04-24T02:00:43.333 INFO:tasks.rados.rados.0.smithi079.stdout:1354: expect (ObjNum 537 snap 121 seq_num 537) 2024-04-24T02:00:43.333 INFO:tasks.rados.rados.0.smithi079.stdout:1355: delete oid 23 current snap is 154 2024-04-24T02:00:43.404 INFO:tasks.rados.rados.0.smithi079.stdout:1354: expect (ObjNum 537 snap 121 seq_num 537) 2024-04-24T02:00:43.527 INFO:tasks.rados.rados.0.smithi079.stdout:1351: done (4 left) 2024-04-24T02:00:43.527 INFO:tasks.rados.rados.0.smithi079.stdout:1352: done (3 left) 2024-04-24T02:00:43.527 INFO:tasks.rados.rados.0.smithi079.stdout:1353: done (2 left) 2024-04-24T02:00:43.527 INFO:tasks.rados.rados.0.smithi079.stdout:1354: done (1 left) 2024-04-24T02:00:43.527 INFO:tasks.rados.rados.0.smithi079.stdout:1355: done (0 left) 2024-04-24T02:00:43.527 INFO:tasks.rados.rados.0.smithi079.stdout:1356: write oid 30 current snap is 154 2024-04-24T02:00:43.527 INFO:tasks.rados.rados.0.smithi079.stdout:1356: seq_num 660 ranges {692620=470698,1755292=752886} 2024-04-24T02:00:43.533 INFO:tasks.rados.rados.0.smithi079.stdout:1356: writing smithi07929762-30 from 692620 to 1163318 tid 1 2024-04-24T02:00:43.540 INFO:tasks.rados.rados.0.smithi079.stdout:1356: writing smithi07929762-30 from 1755292 to 2508178 tid 2 2024-04-24T02:00:43.540 INFO:tasks.rados.rados.0.smithi079.stdout:1357: snap_create 2024-04-24T02:00:43.559 INFO:tasks.rados.rados.0.smithi079.stdout:1356: finishing write tid 1 to smithi07929762-30 2024-04-24T02:00:43.568 INFO:tasks.rados.rados.0.smithi079.stdout:1356: finishing write tid 2 to smithi07929762-30 2024-04-24T02:00:43.572 INFO:tasks.rados.rados.0.smithi079.stdout:1356: finishing write tid 3 to smithi07929762-30 2024-04-24T02:00:43.574 INFO:tasks.rados.rados.0.smithi079.stdout:1356: finishing write tid 4 to smithi07929762-30 2024-04-24T02:00:43.574 INFO:tasks.rados.rados.0.smithi079.stdout:1356: oid 30 updating version 0 to 190 2024-04-24T02:00:43.574 INFO:tasks.rados.rados.0.smithi079.stdout:1356: oid 30 version 190 is already newer than 190 2024-04-24T02:00:43.574 INFO:tasks.rados.rados.0.smithi079.stdout:1356: oid 30 version 190 is already newer than 189 2024-04-24T02:00:43.574 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 30 v 190 (ObjNum 660 snap 154 seq_num 660) dirty exists 2024-04-24T02:00:43.574 INFO:tasks.rados.rados.0.smithi079.stdout:1356: left oid 30 (ObjNum 660 snap 154 seq_num 660) 2024-04-24T02:00:44.246 INFO:tasks.rados.rados.0.smithi079.stdout:1356: done (1 left) 2024-04-24T02:00:44.247 INFO:tasks.rados.rados.0.smithi079.stdout:1357: done (0 left) 2024-04-24T02:00:44.247 INFO:tasks.rados.rados.0.smithi079.stdout:1358: delete oid 16 current snap is 155 2024-04-24T02:00:44.281 INFO:tasks.rados.rados.0.smithi079.stdout:1358: done (0 left) 2024-04-24T02:00:44.281 INFO:tasks.rados.rados.0.smithi079.stdout:1359: snap_create 2024-04-24T02:00:44.447 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T02:00:44.552 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:44.550+0000 7fc6a1e57640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:44.556 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:44.555+0000 7fc6a1e57640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:45.257 INFO:tasks.rados.rados.0.smithi079.stdout:1359: done (0 left) 2024-04-24T02:00:45.257 INFO:tasks.rados.rados.0.smithi079.stdout:1360: write (excl) oid 50 current snap is 156 2024-04-24T02:00:45.257 INFO:tasks.rados.rados.0.smithi079.stdout:1360: seq_num 662 ranges {726234=737549,1988609=756555} 2024-04-24T02:00:45.264 INFO:tasks.rados.rados.0.smithi079.stdout:1360: writing smithi07929762-50 from 726234 to 1463783 tid 1 2024-04-24T02:00:45.273 INFO:tasks.rados.rados.0.smithi079.stdout:1360: writing smithi07929762-50 from 1988609 to 2745164 tid 2 2024-04-24T02:00:45.273 INFO:tasks.rados.rados.0.smithi079.stdout:1361: read oid 37 snap 154 2024-04-24T02:00:45.273 INFO:tasks.rados.rados.0.smithi079.stdout:1361: expect deleted 2024-04-24T02:00:45.273 INFO:tasks.rados.rados.0.smithi079.stdout:1362: read oid 36 snap 151 2024-04-24T02:00:45.273 INFO:tasks.rados.rados.0.smithi079.stdout:1362: expect (ObjNum 646 snap 151 seq_num 646) 2024-04-24T02:00:45.273 INFO:tasks.rados.rados.0.smithi079.stdout:1363: read oid 2 snap 154 2024-04-24T02:00:45.274 INFO:tasks.rados.rados.0.smithi079.stdout:1363: expect (ObjNum 652 snap 153 seq_num 652) 2024-04-24T02:00:45.274 INFO:tasks.rados.rados.0.smithi079.stdout:1364: read oid 15 snap -1 2024-04-24T02:00:45.274 INFO:tasks.rados.rados.0.smithi079.stdout:1364: expect (ObjNum 657 snap 154 seq_num 657) 2024-04-24T02:00:45.274 INFO:tasks.rados.rados.0.smithi079.stdout:1365: read oid 9 snap -1 2024-04-24T02:00:45.274 INFO:tasks.rados.rados.0.smithi079.stdout:1365: expect (ObjNum 577 snap 132 seq_num 577) 2024-04-24T02:00:45.274 INFO:tasks.rados.rados.0.smithi079.stdout:1366: snap_create 2024-04-24T02:00:45.323 INFO:tasks.rados.rados.0.smithi079.stdout:1362: expect (ObjNum 646 snap 151 seq_num 646) 2024-04-24T02:00:45.436 INFO:tasks.rados.rados.0.smithi079.stdout:1365: expect (ObjNum 577 snap 132 seq_num 577) 2024-04-24T02:00:45.667 INFO:tasks.rados.rados.0.smithi079.stdout:1360: finishing write tid 1 to smithi07929762-50 2024-04-24T02:00:45.667 INFO:tasks.rados.rados.0.smithi079.stdout:1363: expect (ObjNum 652 snap 153 seq_num 652) 2024-04-24T02:00:45.825 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-04-24T02:00:45.825 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-04-24T02:00:45.825 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-04-24T02:00:45.826 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-04-24T02:00:45.935 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:45.929+0000 7fd61cb9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:45.939 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:45.938+0000 7fd61cb9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:45.953 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T02:00:45.954 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T02:00:45.954 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T02:00:45.954 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T02:00:45.954 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T02:00:45.954 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T02:00:45.954 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T02:00:45.954 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T02:00:45.954 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T02:00:45.954 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T02:00:45.954 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T02:00:45.954 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T02:00:45.956 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T02:00:45.957 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T02:00:45.957 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-04-24T02:00:45.958 INFO:tasks.rados.rados.0.smithi079.stdout:1364: expect (ObjNum 657 snap 154 seq_num 657) 2024-04-24T02:00:46.067 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:46.061+0000 7fb1be95d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:46.071 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:46.071+0000 7fb1be95d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:46.079 INFO:tasks.rados.rados.0.smithi079.stdout:1360: finishing write tid 2 to smithi07929762-50 2024-04-24T02:00:46.079 INFO:tasks.rados.rados.0.smithi079.stdout:1360: finishing write tid 3 to smithi07929762-50 2024-04-24T02:00:46.079 INFO:tasks.rados.rados.0.smithi079.stdout:1360: finishing write tid 4 to smithi07929762-50 2024-04-24T02:00:46.079 INFO:tasks.rados.rados.0.smithi079.stdout:1360: oid 50 updating version 0 to 302 2024-04-24T02:00:46.079 INFO:tasks.rados.rados.0.smithi079.stdout:1360: oid 50 version 302 is already newer than 300 2024-04-24T02:00:46.079 INFO:tasks.rados.rados.0.smithi079.stdout:1360: oid 50 version 302 is already newer than 302 2024-04-24T02:00:46.079 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 50 v 302 (ObjNum 662 snap 156 seq_num 662) dirty exists 2024-04-24T02:00:46.079 INFO:tasks.rados.rados.0.smithi079.stdout:1360: left oid 50 (ObjNum 662 snap 156 seq_num 662) 2024-04-24T02:00:46.087 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T02:00:46.087 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T02:00:46.087 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T02:00:46.087 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T02:00:46.087 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T02:00:46.087 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T02:00:46.087 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T02:00:46.087 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T02:00:46.087 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T02:00:46.087 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T02:00:46.088 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T02:00:46.088 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T02:00:46.090 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T02:00:46.091 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T02:00:46.091 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-04-24T02:00:46.198 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:46.192+0000 7fa05495d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:46.201 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:46.201+0000 7fa05495d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:46.219 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T02:00:46.219 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T02:00:46.219 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T02:00:46.219 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T02:00:46.219 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T02:00:46.219 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T02:00:46.219 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T02:00:46.219 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T02:00:46.219 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T02:00:46.219 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T02:00:46.219 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T02:00:46.219 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T02:00:46.221 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T02:00:46.222 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T02:00:46.222 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-04-24T02:00:46.255 INFO:tasks.rados.rados.0.smithi079.stdout:1360: done (6 left) 2024-04-24T02:00:46.255 INFO:tasks.rados.rados.0.smithi079.stdout:1361: done (5 left) 2024-04-24T02:00:46.255 INFO:tasks.rados.rados.0.smithi079.stdout:1362: done (4 left) 2024-04-24T02:00:46.255 INFO:tasks.rados.rados.0.smithi079.stdout:1363: done (3 left) 2024-04-24T02:00:46.255 INFO:tasks.rados.rados.0.smithi079.stdout:1364: done (2 left) 2024-04-24T02:00:46.256 INFO:tasks.rados.rados.0.smithi079.stdout:1365: done (1 left) 2024-04-24T02:00:46.256 INFO:tasks.rados.rados.0.smithi079.stdout:1366: done (0 left) 2024-04-24T02:00:46.256 INFO:tasks.rados.rados.0.smithi079.stdout:1367: snap_remove snap 156 2024-04-24T02:00:46.351 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:46.345+0000 7f8b7cd54640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:46.355 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:46.354+0000 7f8b7cd54640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:46.371 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T02:00:46.371 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T02:00:46.371 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T02:00:46.371 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T02:00:46.371 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T02:00:46.371 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T02:00:46.371 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T02:00:46.371 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T02:00:46.371 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T02:00:46.371 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T02:00:46.371 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T02:00:46.371 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T02:00:46.374 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T02:00:46.374 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T02:00:46.649 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T02:00:46.753 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:46.751+0000 7f2f2582d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:46.757 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:46.756+0000 7f2f2582d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:47.259 INFO:tasks.rados.rados.0.smithi079.stdout:1367: done (0 left) 2024-04-24T02:00:47.259 INFO:tasks.rados.rados.0.smithi079.stdout:1368: read oid 40 snap -1 2024-04-24T02:00:47.259 INFO:tasks.rados.rados.0.smithi079.stdout:1368: expect deleted 2024-04-24T02:00:47.259 INFO:tasks.rados.rados.0.smithi079.stdout:1369: snap_create 2024-04-24T02:00:48.260 INFO:tasks.rados.rados.0.smithi079.stdout:1368: done (1 left) 2024-04-24T02:00:48.260 INFO:tasks.rados.rados.0.smithi079.stdout:1369: done (0 left) 2024-04-24T02:00:48.260 INFO:tasks.rados.rados.0.smithi079.stdout:1370: snap_remove snap 154 2024-04-24T02:00:48.851 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T02:00:48.969 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:48.967+0000 7f76f415d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:48.973 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:48.972+0000 7f76f415d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:49.265 INFO:tasks.rados.rados.0.smithi079.stdout:1370: done (0 left) 2024-04-24T02:00:49.265 INFO:tasks.rados.rados.0.smithi079.stdout:1371: snap_create 2024-04-24T02:00:50.267 INFO:tasks.rados.rados.0.smithi079.stdout:1371: done (0 left) 2024-04-24T02:00:50.267 INFO:tasks.rados.rados.0.smithi079.stdout:1372: write (excl) oid 17 current snap is 159 2024-04-24T02:00:50.267 INFO:tasks.rados.rados.0.smithi079.stdout:1372: seq_num 663 ranges {743041=412798,1930738=616566,3040503=102538} 2024-04-24T02:00:50.273 INFO:tasks.rados.rados.0.smithi079.stdout:1372: writing smithi07929762-17 from 743041 to 1155839 tid 1 2024-04-24T02:00:50.281 INFO:tasks.rados.rados.0.smithi079.stdout:1372: writing smithi07929762-17 from 1930738 to 2547304 tid 2 2024-04-24T02:00:50.285 INFO:tasks.rados.rados.0.smithi079.stdout:1372: writing smithi07929762-17 from 3040503 to 3143041 tid 3 2024-04-24T02:00:50.285 INFO:tasks.rados.rados.0.smithi079.stdout:1373: snap_create 2024-04-24T02:00:50.403 INFO:tasks.rados.rados.0.smithi079.stdout:1372: finishing write tid 1 to smithi07929762-17 2024-04-24T02:00:50.433 INFO:tasks.rados.rados.0.smithi079.stdout:1372: finishing write tid 2 to smithi07929762-17 2024-04-24T02:00:50.437 INFO:tasks.rados.rados.0.smithi079.stdout:1372: finishing write tid 3 to smithi07929762-17 2024-04-24T02:00:50.445 INFO:tasks.rados.rados.0.smithi079.stdout:1372: finishing write tid 4 to smithi07929762-17 2024-04-24T02:00:50.446 INFO:tasks.rados.rados.0.smithi079.stdout:1372: finishing write tid 5 to smithi07929762-17 2024-04-24T02:00:50.446 INFO:tasks.rados.rados.0.smithi079.stdout:1372: oid 17 updating version 0 to 359 2024-04-24T02:00:50.446 INFO:tasks.rados.rados.0.smithi079.stdout:1372: oid 17 version 359 is already newer than 359 2024-04-24T02:00:50.446 INFO:tasks.rados.rados.0.smithi079.stdout:1372: oid 17 version 359 is already newer than 359 2024-04-24T02:00:50.446 INFO:tasks.rados.rados.0.smithi079.stdout:1372: oid 17 version 359 is already newer than 356 2024-04-24T02:00:50.446 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 17 v 359 (ObjNum 663 snap 159 seq_num 663) dirty exists 2024-04-24T02:00:50.446 INFO:tasks.rados.rados.0.smithi079.stdout:1372: left oid 17 (ObjNum 663 snap 159 seq_num 663) 2024-04-24T02:00:51.070 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T02:00:51.176 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:51.174+0000 7f66b959d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:51.180 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:51.179+0000 7f66b959d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:51.280 INFO:tasks.rados.rados.0.smithi079.stdout:1372: done (1 left) 2024-04-24T02:00:51.281 INFO:tasks.rados.rados.0.smithi079.stdout:1373: done (0 left) 2024-04-24T02:00:51.281 INFO:tasks.rados.rados.0.smithi079.stdout:1374: delete oid 13 current snap is 160 2024-04-24T02:00:51.332 INFO:tasks.rados.rados.0.smithi079.stdout:1374: done (0 left) 2024-04-24T02:00:51.333 INFO:tasks.rados.rados.0.smithi079.stdout:1375: snap_create 2024-04-24T02:00:51.375 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-04-24T02:00:51.375 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-04-24T02:00:51.376 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-04-24T02:00:51.376 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-04-24T02:00:51.485 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:51.479+0000 7f85b3b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:51.489 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:51.488+0000 7f85b3b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:51.503 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T02:00:51.504 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T02:00:51.504 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T02:00:51.504 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T02:00:51.504 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T02:00:51.504 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T02:00:51.504 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T02:00:51.504 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T02:00:51.504 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T02:00:51.504 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T02:00:51.504 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T02:00:51.504 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T02:00:51.507 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T02:00:51.507 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T02:00:51.507 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-04-24T02:00:51.615 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:51.609+0000 7f04d5e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:51.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:51.618+0000 7f04d5e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:51.633 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T02:00:51.633 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T02:00:51.633 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T02:00:51.633 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T02:00:51.633 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T02:00:51.633 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T02:00:51.633 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T02:00:51.633 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T02:00:51.633 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T02:00:51.634 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T02:00:51.634 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T02:00:51.634 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T02:00:51.636 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T02:00:51.636 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T02:00:51.637 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-04-24T02:00:51.744 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:51.738+0000 7fb085e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:51.748 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:51.747+0000 7fb085e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:51.765 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T02:00:51.765 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T02:00:51.765 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T02:00:51.765 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T02:00:51.765 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T02:00:51.765 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T02:00:51.765 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T02:00:51.765 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T02:00:51.765 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T02:00:51.765 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T02:00:51.765 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T02:00:51.766 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T02:00:51.768 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T02:00:51.768 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T02:00:51.768 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-04-24T02:00:51.877 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:51.871+0000 7fa4c98ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:51.880 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:51.879+0000 7fa4c98ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:51.897 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T02:00:51.897 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T02:00:51.897 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T02:00:51.897 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T02:00:51.897 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T02:00:51.898 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T02:00:51.898 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T02:00:51.898 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T02:00:51.898 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T02:00:51.898 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T02:00:51.898 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T02:00:51.898 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T02:00:51.900 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T02:00:51.901 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T02:00:52.278 INFO:tasks.rados.rados.0.smithi079.stdout:1375: done (0 left) 2024-04-24T02:00:52.278 INFO:tasks.rados.rados.0.smithi079.stdout:1376: snap_remove snap 151 2024-04-24T02:00:53.275 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T02:00:53.282 INFO:tasks.rados.rados.0.smithi079.stdout:1376: done (0 left) 2024-04-24T02:00:53.283 INFO:tasks.rados.rados.0.smithi079.stdout:1377: delete oid 1 current snap is 161 2024-04-24T02:00:53.413 INFO:tasks.rados.rados.0.smithi079.stdout:1377: done (0 left) 2024-04-24T02:00:53.413 INFO:tasks.rados.rados.0.smithi079.stdout:1378: delete oid 38 current snap is 161 2024-04-24T02:00:53.419 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:53.417+0000 7f2d37f5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:53.423 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:53.422+0000 7f2d37f5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:53.507 INFO:tasks.rados.rados.0.smithi079.stdout:1378: done (0 left) 2024-04-24T02:00:53.507 INFO:tasks.rados.rados.0.smithi079.stdout:1379: write (excl) oid 33 current snap is 161 2024-04-24T02:00:53.507 INFO:tasks.rados.rados.0.smithi079.stdout:1379: seq_num 667 ranges {410269=713794,1648313=507551,2917489=292780} 2024-04-24T02:00:53.513 INFO:tasks.rados.rados.0.smithi079.stdout:1379: writing smithi07929762-33 from 410269 to 1124063 tid 1 2024-04-24T02:00:53.519 INFO:tasks.rados.rados.0.smithi079.stdout:1379: writing smithi07929762-33 from 1648313 to 2155864 tid 2 2024-04-24T02:00:53.525 INFO:tasks.rados.rados.0.smithi079.stdout:1379: writing smithi07929762-33 from 2917489 to 3210269 tid 3 2024-04-24T02:00:53.525 INFO:tasks.rados.rados.0.smithi079.stdout:1380: snap_create 2024-04-24T02:00:53.612 INFO:tasks.rados.rados.0.smithi079.stdout:1379: finishing write tid 1 to smithi07929762-33 2024-04-24T02:00:53.631 INFO:tasks.rados.rados.0.smithi079.stdout:1379: finishing write tid 2 to smithi07929762-33 2024-04-24T02:00:53.638 INFO:tasks.rados.rados.0.smithi079.stdout:1379: finishing write tid 3 to smithi07929762-33 2024-04-24T02:00:53.642 INFO:tasks.rados.rados.0.smithi079.stdout:1379: finishing write tid 4 to smithi07929762-33 2024-04-24T02:00:53.644 INFO:tasks.rados.rados.0.smithi079.stdout:1379: finishing write tid 5 to smithi07929762-33 2024-04-24T02:00:53.644 INFO:tasks.rados.rados.0.smithi079.stdout:1379: oid 33 updating version 0 to 421 2024-04-24T02:00:53.644 INFO:tasks.rados.rados.0.smithi079.stdout:1379: oid 33 updating version 421 to 424 2024-04-24T02:00:53.644 INFO:tasks.rados.rados.0.smithi079.stdout:1379: oid 33 version 424 is already newer than 424 2024-04-24T02:00:53.644 INFO:tasks.rados.rados.0.smithi079.stdout:1379: oid 33 version 424 is already newer than 424 2024-04-24T02:00:53.644 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 33 v 424 (ObjNum 667 snap 161 seq_num 667) dirty exists 2024-04-24T02:00:53.644 INFO:tasks.rados.rados.0.smithi079.stdout:1379: left oid 33 (ObjNum 667 snap 161 seq_num 667) 2024-04-24T02:00:54.284 INFO:tasks.rados.rados.0.smithi079.stdout:1379: done (1 left) 2024-04-24T02:00:54.285 INFO:tasks.rados.rados.0.smithi079.stdout:1380: done (0 left) 2024-04-24T02:00:54.285 INFO:tasks.rados.rados.0.smithi079.stdout:1381: read oid 17 snap -1 2024-04-24T02:00:54.285 INFO:tasks.rados.rados.0.smithi079.stdout:1381: expect (ObjNum 663 snap 159 seq_num 663) 2024-04-24T02:00:54.285 INFO:tasks.rados.rados.0.smithi079.stdout:1382: delete oid 23 current snap is 162 2024-04-24T02:00:54.335 INFO:tasks.rados.rados.0.smithi079.stdout:1382: done (1 left) 2024-04-24T02:00:54.335 INFO:tasks.rados.rados.0.smithi079.stdout:1383: write oid 4 current snap is 162 2024-04-24T02:00:54.335 INFO:tasks.rados.rados.0.smithi079.stdout:1383: seq_num 669 ranges {443883=464292,1365277=511220,2572335=671548} 2024-04-24T02:00:54.341 INFO:tasks.rados.rados.0.smithi079.stdout:1383: writing smithi07929762-4 from 443883 to 908175 tid 1 2024-04-24T02:00:54.346 INFO:tasks.rados.rados.0.smithi079.stdout:1383: writing smithi07929762-4 from 1365277 to 1876497 tid 2 2024-04-24T02:00:54.355 INFO:tasks.rados.rados.0.smithi079.stdout:1383: writing smithi07929762-4 from 2572335 to 3243883 tid 3 2024-04-24T02:00:54.355 INFO:tasks.rados.rados.0.smithi079.stdout:1384: delete oid 22 current snap is 162 2024-04-24T02:00:54.355 INFO:tasks.rados.rados.0.smithi079.stdout:1381: expect (ObjNum 663 snap 159 seq_num 663) 2024-04-24T02:00:54.528 INFO:tasks.rados.rados.0.smithi079.stdout:1381: done (2 left) 2024-04-24T02:00:54.528 INFO:tasks.rados.rados.0.smithi079.stdout:1384: done (1 left) 2024-04-24T02:00:54.528 INFO:tasks.rados.rados.0.smithi079.stdout:1385: write oid 32 current snap is 162 2024-04-24T02:00:54.528 INFO:tasks.rados.rados.0.smithi079.stdout:1383: finishing write tid 1 to smithi07929762-4 2024-04-24T02:00:54.528 INFO:tasks.rados.rados.0.smithi079.stdout:1385: seq_num 671 ranges {477497=731143,1998594=514889,3143534=133963} 2024-04-24T02:00:54.534 INFO:tasks.rados.rados.0.smithi079.stdout:1385: writing smithi07929762-32 from 477497 to 1208640 tid 1 2024-04-24T02:00:54.542 INFO:tasks.rados.rados.0.smithi079.stdout:1385: writing smithi07929762-32 from 1998594 to 2513483 tid 2 2024-04-24T02:00:54.548 INFO:tasks.rados.rados.0.smithi079.stdout:1385: writing smithi07929762-32 from 3143534 to 3277497 tid 3 2024-04-24T02:00:54.548 INFO:tasks.rados.rados.0.smithi079.stdout:1386: snap_remove snap 157 2024-04-24T02:00:55.291 INFO:tasks.rados.rados.0.smithi079.stdout:1386: done (2 left) 2024-04-24T02:00:55.291 INFO:tasks.rados.rados.0.smithi079.stdout:1387: read oid 34 snap -1 2024-04-24T02:00:55.291 INFO:tasks.rados.rados.0.smithi079.stdout:1387: expect deleted 2024-04-24T02:00:55.291 INFO:tasks.rados.rados.0.smithi079.stdout:1388: read oid 9 snap -1 2024-04-24T02:00:55.291 INFO:tasks.rados.rados.0.smithi079.stdout:1388: expect (ObjNum 577 snap 132 seq_num 577) 2024-04-24T02:00:55.291 INFO:tasks.rados.rados.0.smithi079.stdout:1389: snap_remove snap 155 2024-04-24T02:00:55.555 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T02:00:55.660 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:55.658+0000 7f027dced640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:55.664 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:55.663+0000 7f027dced640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:56.292 INFO:tasks.rados.rados.0.smithi079.stdout:1389: done (4 left) 2024-04-24T02:00:56.292 INFO:tasks.rados.rados.0.smithi079.stdout:1390: snap_create 2024-04-24T02:00:56.292 INFO:tasks.rados.rados.0.smithi079.stdout:1383: finishing write tid 2 to smithi07929762-4 2024-04-24T02:00:56.292 INFO:tasks.rados.rados.0.smithi079.stdout:1383: finishing write tid 3 to smithi07929762-4 2024-04-24T02:00:56.292 INFO:tasks.rados.rados.0.smithi079.stdout:1383: finishing write tid 4 to smithi07929762-4 2024-04-24T02:00:56.292 INFO:tasks.rados.rados.0.smithi079.stdout:1383: finishing write tid 5 to smithi07929762-4 2024-04-24T02:00:56.292 INFO:tasks.rados.rados.0.smithi079.stdout:1383: oid 4 updating version 0 to 141 2024-04-24T02:00:56.292 INFO:tasks.rados.rados.0.smithi079.stdout:1383: oid 4 updating version 141 to 142 2024-04-24T02:00:56.292 INFO:tasks.rados.rados.0.smithi079.stdout:1383: oid 4 version 142 is already newer than 142 2024-04-24T02:00:56.292 INFO:tasks.rados.rados.0.smithi079.stdout:1383: oid 4 version 142 is already newer than 142 2024-04-24T02:00:56.293 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 4 v 142 (ObjNum 669 snap 162 seq_num 669) dirty exists 2024-04-24T02:00:56.293 INFO:tasks.rados.rados.0.smithi079.stdout:1383: left oid 4 (ObjNum 669 snap 162 seq_num 669) 2024-04-24T02:00:56.293 INFO:tasks.rados.rados.0.smithi079.stdout:1385: finishing write tid 1 to smithi07929762-32 2024-04-24T02:00:56.293 INFO:tasks.rados.rados.0.smithi079.stdout:1385: finishing write tid 2 to smithi07929762-32 2024-04-24T02:00:56.293 INFO:tasks.rados.rados.0.smithi079.stdout:1385: finishing write tid 3 to smithi07929762-32 2024-04-24T02:00:56.293 INFO:tasks.rados.rados.0.smithi079.stdout:1385: finishing write tid 4 to smithi07929762-32 2024-04-24T02:00:56.293 INFO:tasks.rados.rados.0.smithi079.stdout:1385: finishing write tid 5 to smithi07929762-32 2024-04-24T02:00:56.293 INFO:tasks.rados.rados.0.smithi079.stdout:1385: oid 32 updating version 0 to 247 2024-04-24T02:00:56.293 INFO:tasks.rados.rados.0.smithi079.stdout:1385: oid 32 version 247 is already newer than 247 2024-04-24T02:00:56.293 INFO:tasks.rados.rados.0.smithi079.stdout:1385: oid 32 version 247 is already newer than 247 2024-04-24T02:00:56.293 INFO:tasks.rados.rados.0.smithi079.stdout:1385: oid 32 version 247 is already newer than 247 2024-04-24T02:00:56.293 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 32 v 247 (ObjNum 671 snap 162 seq_num 671) dirty exists 2024-04-24T02:00:56.293 INFO:tasks.rados.rados.0.smithi079.stdout:1385: left oid 32 (ObjNum 671 snap 162 seq_num 671) 2024-04-24T02:00:56.293 INFO:tasks.rados.rados.0.smithi079.stdout:1388: expect (ObjNum 577 snap 132 seq_num 577) 2024-04-24T02:00:56.902 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-04-24T02:00:56.902 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-04-24T02:00:56.902 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-04-24T02:00:56.902 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-04-24T02:00:57.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:57.006+0000 7ff22c19d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:57.015 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:57.015+0000 7ff22c19d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:57.030 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T02:00:57.030 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T02:00:57.030 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T02:00:57.030 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T02:00:57.030 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T02:00:57.030 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T02:00:57.030 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T02:00:57.030 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T02:00:57.030 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T02:00:57.030 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T02:00:57.030 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T02:00:57.030 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T02:00:57.032 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T02:00:57.033 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T02:00:57.033 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-04-24T02:00:57.141 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:57.135+0000 7f2cddaad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:57.144 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:57.144+0000 7f2cddaad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:57.159 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T02:00:57.159 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T02:00:57.159 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T02:00:57.159 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T02:00:57.159 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T02:00:57.159 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T02:00:57.159 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T02:00:57.159 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T02:00:57.159 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T02:00:57.159 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T02:00:57.159 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T02:00:57.160 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T02:00:57.162 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T02:00:57.162 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T02:00:57.162 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-04-24T02:00:57.270 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:57.264+0000 7f4862464640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:57.273 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:57.273+0000 7f4862464640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:57.291 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T02:00:57.291 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T02:00:57.291 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T02:00:57.291 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T02:00:57.291 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T02:00:57.291 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T02:00:57.291 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T02:00:57.292 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T02:00:57.292 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T02:00:57.292 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T02:00:57.292 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T02:00:57.292 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T02:00:57.294 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T02:00:57.294 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T02:00:57.295 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-04-24T02:00:57.303 INFO:tasks.rados.rados.0.smithi079.stdout:1383: done (4 left) 2024-04-24T02:00:57.303 INFO:tasks.rados.rados.0.smithi079.stdout:1385: done (3 left) 2024-04-24T02:00:57.303 INFO:tasks.rados.rados.0.smithi079.stdout:1387: done (2 left) 2024-04-24T02:00:57.303 INFO:tasks.rados.rados.0.smithi079.stdout:1388: done (1 left) 2024-04-24T02:00:57.303 INFO:tasks.rados.rados.0.smithi079.stdout:1390: done (0 left) 2024-04-24T02:00:57.303 INFO:tasks.rados.rados.0.smithi079.stdout:1391: read oid 43 snap -1 2024-04-24T02:00:57.304 INFO:tasks.rados.rados.0.smithi079.stdout:1391: expect (ObjNum 623 snap 144 seq_num 623) 2024-04-24T02:00:57.304 INFO:tasks.rados.rados.0.smithi079.stdout:1392: read oid 5 snap -1 2024-04-24T02:00:57.304 INFO:tasks.rados.rados.0.smithi079.stdout:1392: expect (ObjNum 559 snap 128 seq_num 559) 2024-04-24T02:00:57.304 INFO:tasks.rados.rados.0.smithi079.stdout:1393: read oid 48 snap -1 2024-04-24T02:00:57.304 INFO:tasks.rados.rados.0.smithi079.stdout:1393: expect (ObjNum 609 snap 142 seq_num 609) 2024-04-24T02:00:57.304 INFO:tasks.rados.rados.0.smithi079.stdout:1394: delete oid 16 current snap is 163 2024-04-24T02:00:57.359 INFO:tasks.rados.rados.0.smithi079.stdout:1391: expect (ObjNum 623 snap 144 seq_num 623) 2024-04-24T02:00:57.424 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:57.418+0000 7fe99522d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:57.428 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:57.427+0000 7fe99522d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:57.446 INFO:teuthology.orchestra.run.smithi079.stderr:no valid command found; 10 closest matches: 2024-04-24T02:00:57.446 INFO:teuthology.orchestra.run.smithi079.stderr:config get 2024-04-24T02:00:57.446 INFO:teuthology.orchestra.run.smithi079.stderr:config help 2024-04-24T02:00:57.446 INFO:teuthology.orchestra.run.smithi079.stderr:config set ... 2024-04-24T02:00:57.446 INFO:teuthology.orchestra.run.smithi079.stderr:config show 2024-04-24T02:00:57.446 INFO:teuthology.orchestra.run.smithi079.stderr:deep_scrub 2024-04-24T02:00:57.446 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_ops 2024-04-24T02:00:57.446 INFO:teuthology.orchestra.run.smithi079.stderr:dump_historic_slow_ops 2024-04-24T02:00:57.447 INFO:teuthology.orchestra.run.smithi079.stderr:dump_metrics [] 2024-04-24T02:00:57.447 INFO:teuthology.orchestra.run.smithi079.stderr:dump_ops_in_flight 2024-04-24T02:00:57.447 INFO:teuthology.orchestra.run.smithi079.stderr:dump_pgstate_history 2024-04-24T02:00:57.447 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: invalid command 2024-04-24T02:00:57.449 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T02:00:57.449 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T02:00:57.521 INFO:tasks.rados.rados.0.smithi079.stdout:1392: expect (ObjNum 559 snap 128 seq_num 559) 2024-04-24T02:00:57.647 INFO:tasks.rados.rados.0.smithi079.stdout:1393: expect (ObjNum 609 snap 142 seq_num 609) 2024-04-24T02:00:57.760 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T02:00:57.782 INFO:tasks.rados.rados.0.smithi079.stdout:1391: done (3 left) 2024-04-24T02:00:57.782 INFO:tasks.rados.rados.0.smithi079.stdout:1392: done (2 left) 2024-04-24T02:00:57.782 INFO:tasks.rados.rados.0.smithi079.stdout:1393: done (1 left) 2024-04-24T02:00:57.782 INFO:tasks.rados.rados.0.smithi079.stdout:1394: done (0 left) 2024-04-24T02:00:57.782 INFO:tasks.rados.rados.0.smithi079.stdout:1395: snap_remove snap 161 2024-04-24T02:00:57.863 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:57.861+0000 7f620b4ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:57.867 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:00:57.866+0000 7f620b4ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:00:58.306 INFO:tasks.rados.rados.0.smithi079.stdout:1395: done (0 left) 2024-04-24T02:00:58.306 INFO:tasks.rados.rados.0.smithi079.stdout:1396: read oid 16 snap -1 2024-04-24T02:00:58.306 INFO:tasks.rados.rados.0.smithi079.stdout:1396: expect deleted 2024-04-24T02:00:58.306 INFO:tasks.rados.rados.0.smithi079.stdout:1397: read oid 3 snap -1 2024-04-24T02:00:58.306 INFO:tasks.rados.rados.0.smithi079.stdout:1397: expect (ObjNum 645 snap 151 seq_num 645) 2024-04-24T02:00:58.306 INFO:tasks.rados.rados.0.smithi079.stdout:1398: write (excl) oid 46 current snap is 163 2024-04-24T02:00:58.306 INFO:tasks.rados.rados.0.smithi079.stdout:1398: seq_num 673 ranges {511111=481641,1599205=518558,2682027=489947} 2024-04-24T02:00:58.311 INFO:tasks.rados.rados.0.smithi079.stdout:1398: writing smithi07929762-46 from 511111 to 992752 tid 1 2024-04-24T02:00:58.317 INFO:tasks.rados.rados.0.smithi079.stdout:1398: writing smithi07929762-46 from 1599205 to 2117763 tid 2 2024-04-24T02:00:58.324 INFO:tasks.rados.rados.0.smithi079.stdout:1398: writing smithi07929762-46 from 2682027 to 3171974 tid 3 2024-04-24T02:00:58.324 INFO:tasks.rados.rados.0.smithi079.stdout:1399: write (excl) oid 41 current snap is 163 2024-04-24T02:00:58.324 INFO:tasks.rados.rados.0.smithi079.stdout:1399: seq_num 674 ranges {527918=556890,1657687=778569,3225803=102115} 2024-04-24T02:00:58.330 INFO:tasks.rados.rados.0.smithi079.stdout:1399: writing smithi07929762-41 from 527918 to 1084808 tid 1 2024-04-24T02:00:58.338 INFO:tasks.rados.rados.0.smithi079.stdout:1399: writing smithi07929762-41 from 1657687 to 2436256 tid 2 2024-04-24T02:00:58.343 INFO:tasks.rados.rados.0.smithi079.stdout:1399: writing smithi07929762-41 from 3225803 to 3327918 tid 3 2024-04-24T02:00:58.343 INFO:tasks.rados.rados.0.smithi079.stdout:1400: read oid 35 snap 158 2024-04-24T02:00:58.344 INFO:tasks.rados.rados.0.smithi079.stdout:1400: expect (ObjNum 572 snap 131 seq_num 572) 2024-04-24T02:00:58.344 INFO:tasks.rados.rados.0.smithi079.stdout:1401: read oid 34 snap 162 2024-04-24T02:00:58.344 INFO:tasks.rados.rados.0.smithi079.stdout:1401: expect deleted 2024-04-24T02:00:58.344 INFO:tasks.rados.rados.0.smithi079.stdout:1402: write oid 9 current snap is 163 2024-04-24T02:00:58.344 INFO:tasks.rados.rados.0.smithi079.stdout:1402: seq_num 675 ranges {544725=632139,1716169=522227,2736873=512491} 2024-04-24T02:00:58.350 INFO:tasks.rados.rados.0.smithi079.stdout:1402: writing smithi07929762-9 from 544725 to 1176864 tid 1 2024-04-24T02:00:58.356 INFO:tasks.rados.rados.0.smithi079.stdout:1402: writing smithi07929762-9 from 1716169 to 2238396 tid 2 2024-04-24T02:00:58.362 INFO:tasks.rados.rados.0.smithi079.stdout:1402: writing smithi07929762-9 from 2736873 to 3249364 tid 3 2024-04-24T02:00:58.362 INFO:tasks.rados.rados.0.smithi079.stdout:1403: read oid 15 snap -1 2024-04-24T02:00:58.363 INFO:tasks.rados.rados.0.smithi079.stdout:1403: expect (ObjNum 657 snap 154 seq_num 657) 2024-04-24T02:00:58.363 INFO:tasks.rados.rados.0.smithi079.stdout:1404: write (excl) oid 14 current snap is 163 2024-04-24T02:00:58.363 INFO:tasks.rados.rados.0.smithi079.stdout:1404: seq_num 676 ranges {561532=707388,2058298=782238} 2024-04-24T02:00:58.370 INFO:tasks.rados.rados.0.smithi079.stdout:1404: writing smithi07929762-14 from 561532 to 1268920 tid 1 2024-04-24T02:00:58.383 INFO:tasks.rados.rados.0.smithi079.stdout:1404: writing smithi07929762-14 from 2058298 to 2840536 tid 2 2024-04-24T02:00:58.383 INFO:tasks.rados.rados.0.smithi079.stdout:1405: write oid 24 current snap is 163 2024-04-24T02:00:58.383 INFO:tasks.rados.rados.0.smithi079.stdout:1405: seq_num 677 ranges {578339=498990,1833133=525896,2791719=586620} 2024-04-24T02:00:58.390 INFO:tasks.rados.rados.0.smithi079.stdout:1405: writing smithi07929762-24 from 578339 to 1077329 tid 1 2024-04-24T02:00:58.399 INFO:tasks.rados.rados.0.smithi079.stdout:1405: writing smithi07929762-24 from 1833133 to 2359029 tid 2 2024-04-24T02:00:58.405 INFO:tasks.rados.rados.0.smithi079.stdout:1405: writing smithi07929762-24 from 2791719 to 3378339 tid 3 2024-04-24T02:00:58.405 INFO:tasks.rados.rados.0.smithi079.stdout:1406: write (excl) oid 44 current snap is 163 2024-04-24T02:00:58.405 INFO:tasks.rados.rados.0.smithi079.stdout:1406: seq_num 678 ranges {595146=574239,1891615=669554,3219142=176004} 2024-04-24T02:00:58.414 INFO:tasks.rados.rados.0.smithi079.stdout:1406: writing smithi07929762-44 from 595146 to 1169385 tid 1 2024-04-24T02:00:58.425 INFO:tasks.rados.rados.0.smithi079.stdout:1406: writing smithi07929762-44 from 1891615 to 2561169 tid 2 2024-04-24T02:00:58.431 INFO:tasks.rados.rados.0.smithi079.stdout:1406: writing smithi07929762-44 from 3219142 to 3395146 tid 3 2024-04-24T02:00:58.431 INFO:tasks.rados.rados.0.smithi079.stdout:1407: write oid 1 current snap is 163 2024-04-24T02:00:58.431 INFO:tasks.rados.rados.0.smithi079.stdout:1407: seq_num 679 ranges {611953=649488,1950097=529565,3246565=165388} 2024-04-24T02:00:58.439 INFO:tasks.rados.rados.0.smithi079.stdout:1407: writing smithi07929762-1 from 611953 to 1261441 tid 1 2024-04-24T02:00:58.449 INFO:tasks.rados.rados.0.smithi079.stdout:1407: writing smithi07929762-1 from 1950097 to 2479662 tid 2 2024-04-24T02:00:58.454 INFO:tasks.rados.rados.0.smithi079.stdout:1407: writing smithi07929762-1 from 3246565 to 3411953 tid 3 2024-04-24T02:00:58.454 INFO:tasks.rados.rados.0.smithi079.stdout:1400: expect (ObjNum 572 snap 131 seq_num 572) 2024-04-24T02:00:58.766 INFO:tasks.rados.rados.0.smithi079.stdout:1396: done (11 left) 2024-04-24T02:00:58.766 INFO:tasks.rados.rados.0.smithi079.stdout:1400: done (10 left) 2024-04-24T02:00:58.766 INFO:tasks.rados.rados.0.smithi079.stdout:1408: read oid 38 snap -1 2024-04-24T02:00:58.766 INFO:tasks.rados.rados.0.smithi079.stdout:1408: expect deleted 2024-04-24T02:00:58.766 INFO:tasks.rados.rados.0.smithi079.stdout:1409: write (excl) oid 19 current snap is 163 2024-04-24T02:00:58.766 INFO:tasks.rados.rados.0.smithi079.stdout:1409: seq_num 680 ranges {628760=724737,1892226=673223,3157635=271125} 2024-04-24T02:00:58.772 INFO:tasks.rados.rados.0.smithi079.stdout:1409: writing smithi07929762-19 from 628760 to 1353497 tid 1 2024-04-24T02:00:58.779 INFO:tasks.rados.rados.0.smithi079.stdout:1409: writing smithi07929762-19 from 1892226 to 2565449 tid 2 2024-04-24T02:00:58.784 INFO:tasks.rados.rados.0.smithi079.stdout:1409: writing smithi07929762-19 from 3157635 to 3428760 tid 3 2024-04-24T02:00:58.784 INFO:tasks.rados.rados.0.smithi079.stdout:1410: write oid 2 current snap is 163 2024-04-24T02:00:58.784 INFO:tasks.rados.rados.0.smithi079.stdout:1410: seq_num 681 ranges {645567=799986,1950708=533234,3185058=260509} 2024-04-24T02:00:58.794 INFO:tasks.rados.rados.0.smithi079.stdout:1410: writing smithi07929762-2 from 645567 to 1445553 tid 1 2024-04-24T02:00:58.800 INFO:tasks.rados.rados.0.smithi079.stdout:1410: writing smithi07929762-2 from 1950708 to 2483942 tid 2 2024-04-24T02:00:58.807 INFO:tasks.rados.rados.0.smithi079.stdout:1410: writing smithi07929762-2 from 3185058 to 3445567 tid 3 2024-04-24T02:00:58.807 INFO:tasks.rados.rados.0.smithi079.stdout:1411: snap_remove snap 159 2024-04-24T02:00:59.310 INFO:tasks.rados.rados.0.smithi079.stdout:1411: done (13 left) 2024-04-24T02:00:59.310 INFO:tasks.rados.rados.0.smithi079.stdout:1412: snap_create 2024-04-24T02:00:59.310 INFO:tasks.rados.rados.0.smithi079.stdout:1403: expect (ObjNum 657 snap 154 seq_num 657) 2024-04-24T02:00:59.428 INFO:tasks.rados.rados.0.smithi079.stdout:1397: expect (ObjNum 645 snap 151 seq_num 645) 2024-04-24T02:00:59.573 INFO:tasks.rados.rados.0.smithi079.stdout:1398: finishing write tid 1 to smithi07929762-46 2024-04-24T02:00:59.573 INFO:tasks.rados.rados.0.smithi079.stdout:1399: finishing write tid 1 to smithi07929762-41 2024-04-24T02:00:59.573 INFO:tasks.rados.rados.0.smithi079.stdout:1406: finishing write tid 1 to smithi07929762-44 2024-04-24T02:00:59.573 INFO:tasks.rados.rados.0.smithi079.stdout:1407: finishing write tid 1 to smithi07929762-1 2024-04-24T02:00:59.573 INFO:tasks.rados.rados.0.smithi079.stdout:1407: finishing write tid 2 to smithi07929762-1 2024-04-24T02:00:59.573 INFO:tasks.rados.rados.0.smithi079.stdout:1407: finishing write tid 3 to smithi07929762-1 2024-04-24T02:00:59.573 INFO:tasks.rados.rados.0.smithi079.stdout:1404: finishing write tid 1 to smithi07929762-14 2024-04-24T02:00:59.573 INFO:tasks.rados.rados.0.smithi079.stdout:1402: finishing write tid 1 to smithi07929762-9 2024-04-24T02:00:59.573 INFO:tasks.rados.rados.0.smithi079.stdout:1407: finishing write tid 4 to smithi07929762-1 2024-04-24T02:00:59.573 INFO:tasks.rados.rados.0.smithi079.stdout:1398: finishing write tid 3 to smithi07929762-46 2024-04-24T02:00:59.574 INFO:tasks.rados.rados.0.smithi079.stdout:1407: finishing write tid 5 to smithi07929762-1 2024-04-24T02:00:59.574 INFO:tasks.rados.rados.0.smithi079.stdout:1407: oid 1 updating version 0 to 636 2024-04-24T02:00:59.574 INFO:tasks.rados.rados.0.smithi079.stdout:1407: oid 1 version 636 is already newer than 636 2024-04-24T02:00:59.574 INFO:tasks.rados.rados.0.smithi079.stdout:1407: oid 1 version 636 is already newer than 636 2024-04-24T02:00:59.574 INFO:tasks.rados.rados.0.smithi079.stdout:1407: oid 1 version 636 is already newer than 636 2024-04-24T02:00:59.574 INFO:tasks.rados.rados.0.smithi079.stdout:update_object_version oid 1 v 636 (ObjNum 679 snap 163 seq_num 679) dirty exists 2024-04-24T02:00:59.574 INFO:tasks.rados.rados.0.smithi079.stdout:1407: left oid 1 (ObjNum 679 snap 163 seq_num 679) 2024-04-24T02:00:59.574 INFO:tasks.rados.rados.0.smithi079.stdout:1402: finishing write tid 2 to smithi07929762-9 2024-04-24T02:00:59.574 INFO:tasks.rados.rados.0.smithi079.stdout:1405: finishing write tid 1 to smithi07929762-24 2024-04-24T02:00:59.574 INFO:tasks.rados.rados.0.smithi079.stdout:1398: finishing write tid 4 to smithi07929762-46 2024-04-24T02:00:59.574 INFO:tasks.rados.rados.0.smithi079.stdout:1406: finishing write tid 2 to smithi07929762-44 2024-04-24T02:00:59.574 INFO:tasks.rados.rados.0.smithi079.stdout:1405: finishing write tid 2 to smithi07929762-24 2024-04-24T02:00:59.574 INFO:tasks.rados.rados.0.smithi079.stdout:1402: finishing write tid 3 to smithi07929762-9 2024-04-24T02:00:59.574 INFO:tasks.rados.rados.0.smithi079.stdout:1398: finishing write tid 2 to smithi07929762-46 2024-04-24T02:00:59.575 INFO:tasks.rados.rados.0.smithi079.stderr:Error: finished tid 2 when last_acked_tid was 4 2024-04-24T02:00:59.575 INFO:tasks.rados.rados.0.smithi079.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-3227-g424d8c43/rpm/el9/BUILD/ceph-19.0.0-3227-g424d8c43/src/test/osd/RadosModel.h: In function 'virtual void WriteOp::_finish(TestOp::CallbackInfo*)' thread 7f5c9effd640 time 2024-04-24T02:00:59.573481+0000 2024-04-24T02:00:59.575 INFO:tasks.rados.rados.0.smithi079.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-3227-g424d8c43/rpm/el9/BUILD/ceph-19.0.0-3227-g424d8c43/src/test/osd/RadosModel.h: 1041: ceph_abort_msg("abort() called") 2024-04-24T02:00:59.575 INFO:tasks.rados.rados.0.smithi079.stderr: ceph version 19.0.0-3227-g424d8c43 (424d8c4389b684b0a4d72510e1b4d398d8e6c8f4) squid (dev) 2024-04-24T02:00:59.575 INFO:tasks.rados.rados.0.smithi079.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcc) [0x7f5cb3e4708a] 2024-04-24T02:00:59.575 INFO:tasks.rados.rados.0.smithi079.stderr: 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x1bc) [0x44f17a] 2024-04-24T02:00:59.575 INFO:tasks.rados.rados.0.smithi079.stderr: 3: (TestOp::finish(TestOp::CallbackInfo*)+0xe) [0x460e40] 2024-04-24T02:00:59.576 INFO:tasks.rados.rados.0.smithi079.stderr: 4: (write_callback(void*, void*)+0x1c) [0x460e7b] 2024-04-24T02:00:59.576 INFO:tasks.rados.rados.0.smithi079.stderr: 5: /lib64/librados.so.2(+0x95d9a) [0x7f5cb48b0d9a] 2024-04-24T02:00:59.576 INFO:tasks.rados.rados.0.smithi079.stderr: 6: /lib64/librados.so.2(+0x9fa94) [0x7f5cb48baa94] 2024-04-24T02:00:59.576 INFO:tasks.rados.rados.0.smithi079.stderr: 7: /lib64/librados.so.2(+0x94dc9) [0x7f5cb48afdc9] 2024-04-24T02:00:59.576 INFO:tasks.rados.rados.0.smithi079.stderr: 8: /lib64/librados.so.2(+0xa12c2) [0x7f5cb48bc2c2] 2024-04-24T02:00:59.576 INFO:tasks.rados.rados.0.smithi079.stderr: 9: /lib64/librados.so.2(+0xa1cbe) [0x7f5cb48bccbe] 2024-04-24T02:00:59.576 INFO:tasks.rados.rados.0.smithi079.stderr: 10: /lib64/librados.so.2(+0xadc67) [0x7f5cb48c8c67] 2024-04-24T02:00:59.576 INFO:tasks.rados.rados.0.smithi079.stderr: 11: /lib64/librados.so.2(+0xae46d) [0x7f5cb48c946d] 2024-04-24T02:00:59.576 INFO:tasks.rados.rados.0.smithi079.stderr: 12: /lib64/librados.so.2(+0xb0a6f) [0x7f5cb48cba6f] 2024-04-24T02:00:59.576 INFO:tasks.rados.rados.0.smithi079.stderr: 13: /lib64/librados.so.2(+0xb0ae8) [0x7f5cb48cbae8] 2024-04-24T02:00:59.576 INFO:tasks.rados.rados.0.smithi079.stderr: 14: /lib64/librados.so.2(+0xb0b05) [0x7f5cb48cbb05] 2024-04-24T02:00:59.576 INFO:tasks.rados.rados.0.smithi079.stderr: 15: /lib64/libstdc++.so.6(+0xdbad4) [0x7f5cb4adbad4] 2024-04-24T02:00:59.576 INFO:tasks.rados.rados.0.smithi079.stderr: 16: /lib64/libc.so.6(+0x89c02) [0x7f5cb2c89c02] 2024-04-24T02:00:59.576 INFO:tasks.rados.rados.0.smithi079.stderr: 17: /lib64/libc.so.6(+0x10ec40) [0x7f5cb2d0ec40] 2024-04-24T02:00:59.964 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T02:01:00.073 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:01:00.071+0000 7ff71dc6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:01:00.077 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:01:00.076+0000 7ff71dc6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:01:01.145 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T02:01:01.145 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_ceph-c_424d8c4389b684b0a4d72510e1b4d398d8e6c8f4/qa/tasks/rados.py", line 289, in thread run.wait(tests.values()) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 179, in _raise_for_status raise CommandCrashedError(command=self.command) teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 0 --op copy_from 0 --op write_excl 50 --pool unique_pool_0' 2024-04-24T02:01:01.147 ERROR:teuthology.run_tasks:Manager failed: rados Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_424d8c4389b684b0a4d72510e1b4d398d8e6c8f4/qa/tasks/rados.py", line 308, in task running.get() File "src/gevent/greenlet.py", line 831, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/gevent/_compat.py", line 49, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_ceph-c_424d8c4389b684b0a4d72510e1b4d398d8e6c8f4/qa/tasks/rados.py", line 289, in thread run.wait(tests.values()) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 179, in _raise_for_status raise CommandCrashedError(command=self.command) teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 0 --op copy_from 0 --op write_excl 50 --pool unique_pool_0' 2024-04-24T02:01:01.148 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-04-24T02:01:01.157 INFO:tasks.thrashosds:joining thrashosds 2024-04-24T02:01:02.450 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-04-24T02:01:02.560 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:01:02.558+0000 7f8748cad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:01:02.564 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:01:02.563+0000 7f8748cad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:01:02.578 INFO:teuthology.orchestra.run.smithi079.stdout:{} 2024-04-24T02:01:02.579 INFO:teuthology.orchestra.run.smithi079.stderr:mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-24T02:01:02.589 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-04-24T02:01:02.697 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:01:02.690+0000 7f2d85b5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:01:02.701 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:01:02.700+0000 7f2d85b5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:01:03.626 INFO:teuthology.orchestra.run.smithi079.stderr:reweighted osd.0 to 1 (10000) 2024-04-24T02:01:03.644 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 1 2024-04-24T02:01:03.756 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:01:03.749+0000 7f34f166d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:01:03.760 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:01:03.760+0000 7f34f166d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:01:04.627 INFO:teuthology.orchestra.run.smithi079.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-04-24T02:01:04.643 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-04-24T02:01:04.754 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:01:04.747+0000 7fe8fd66d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:01:04.758 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:01:04.757+0000 7fe8fd66d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:01:05.646 INFO:teuthology.orchestra.run.smithi079.stderr:reweighted osd.1 to 1 (10000) 2024-04-24T02:01:05.662 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 2024-04-24T02:01:05.772 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:01:05.766+0000 7fb1d919d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:01:05.776 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:01:05.775+0000 7fb1d919d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:01:06.647 INFO:teuthology.orchestra.run.smithi079.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-04-24T02:01:06.662 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-04-24T02:01:06.771 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:01:06.764+0000 7f359186d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:01:06.774 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:01:06.774+0000 7f359186d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:01:07.655 INFO:teuthology.orchestra.run.smithi079.stderr:reweighted osd.2 to 1 (10000) 2024-04-24T02:01:07.669 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 1 2024-04-24T02:01:07.780 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:01:07.773+0000 7fd63de2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:01:07.784 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:01:07.783+0000 7fd63de2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:01:08.656 INFO:teuthology.orchestra.run.smithi079.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-04-24T02:01:08.672 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-04-24T02:01:08.784 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:01:08.778+0000 7f28f90ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:01:08.788 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:01:08.787+0000 7f28f90ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:01:09.663 INFO:teuthology.orchestra.run.smithi079.stderr:reweighted osd.3 to 1 (10000) 2024-04-24T02:01:09.679 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 1 2024-04-24T02:01:09.790 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:01:09.783+0000 7fdccb42d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:01:09.793 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:01:09.793+0000 7fdccb42d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:01:10.664 INFO:teuthology.orchestra.run.smithi079.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-04-24T02:01:10.678 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-04-24T02:01:10.678 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-04-24T02:01:10.678 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T02:01:10.785 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:01:10.780+0000 7f79ea75d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:01:10.789 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:01:10.788+0000 7f79ea75d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:01:10.978 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T02:01:10.978 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":351,"fsid":"e600dd1c-0bef-4ea0-91c9-2f649f75622f","created":"2024-04-24T01:55:06.017534+0000","modified":"2024-04-24T02:01:10.663781+0000","last_up_change":"2024-04-24T01:55:14.120010+0000","last_in_change":"2024-04-24T01:55:08.126530+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T01:55:14.952331+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.75,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-24T01:55:16.243536+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"unique_pool_0","create_time":"2024-04-24T01:55:28.358480+0000","flags":270369,"flags_names":"hashpspool,nopgchange,selfmanaged_snaps,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"343","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":165,"snap_epoch":343,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1,"score_stable":1,"optimal_score":1,"raw_score_acting":1,"raw_score_stable":1,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1b20709f-f1e7-4d59-8402-0d13620f747e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6800","nonce":3830851766}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6803","nonce":3830851766}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6805","nonce":3830851766}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":3830851766}]},"public_addr":"172.21.15.79:6800/3830851766","cluster_addr":"172.21.15.79:6803/3830851766","heartbeat_back_addr":"172.21.15.79:6805/3830851766","heartbeat_front_addr":"172.21.15.79:6808/3830851766","state":["exists","up"]},{"osd":1,"uuid":"54fd0eee-6d68-4c92-9ac7-6b21c87783f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6801","nonce":1823728464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":1823728464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":1823728464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6807","nonce":1823728464}]},"public_addr":"172.21.15.79:6801/1823728464","cluster_addr":"172.21.15.79:6802/1823728464","heartbeat_back_addr":"172.21.15.79:6806/1823728464","heartbeat_front_addr":"172.21.15.79:6807/1823728464","state":["exists","up"]},{"osd":2,"uuid":"35820c96-5f2a-4e2e-bc4b-714ab8e1c841","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6803","nonce":2585849279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":2585849279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":2585849279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6817","nonce":2585849279}]},"public_addr":"172.21.15.103:6803/2585849279","cluster_addr":"172.21.15.103:6804/2585849279","heartbeat_back_addr":"172.21.15.103:6816/2585849279","heartbeat_front_addr":"172.21.15.103:6817/2585849279","state":["exists","up"]},{"osd":3,"uuid":"da975ea1-eeb6-4619-b73f-786b46dd8fe6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6809","nonce":659350107}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":659350107}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":659350107}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6811","nonce":659350107}]},"public_addr":"172.21.15.103:6809/659350107","cluster_addr":"172.21.15.103:6808/659350107","heartbeat_back_addr":"172.21.15.103:6812/659350107","heartbeat_front_addr":"172.21.15.103:6811/659350107","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":158},{"begin":161,"length":1},{"begin":163,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T02:01:10.992 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-04-24T02:01:10.992 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T02:01:11.099 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:01:11.094+0000 7ff634a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:01:11.103 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:01:11.102+0000 7ff634a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:01:11.289 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T02:01:11.289 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":351,"fsid":"e600dd1c-0bef-4ea0-91c9-2f649f75622f","created":"2024-04-24T01:55:06.017534+0000","modified":"2024-04-24T02:01:10.663781+0000","last_up_change":"2024-04-24T01:55:14.120010+0000","last_in_change":"2024-04-24T01:55:08.126530+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T01:55:14.952331+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.75,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-24T01:55:16.243536+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"unique_pool_0","create_time":"2024-04-24T01:55:28.358480+0000","flags":270369,"flags_names":"hashpspool,nopgchange,selfmanaged_snaps,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"343","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":165,"snap_epoch":343,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1,"score_stable":1,"optimal_score":1,"raw_score_acting":1,"raw_score_stable":1,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1b20709f-f1e7-4d59-8402-0d13620f747e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6800","nonce":3830851766}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6803","nonce":3830851766}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6805","nonce":3830851766}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":3830851766}]},"public_addr":"172.21.15.79:6800/3830851766","cluster_addr":"172.21.15.79:6803/3830851766","heartbeat_back_addr":"172.21.15.79:6805/3830851766","heartbeat_front_addr":"172.21.15.79:6808/3830851766","state":["exists","up"]},{"osd":1,"uuid":"54fd0eee-6d68-4c92-9ac7-6b21c87783f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6801","nonce":1823728464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":1823728464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":1823728464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6807","nonce":1823728464}]},"public_addr":"172.21.15.79:6801/1823728464","cluster_addr":"172.21.15.79:6802/1823728464","heartbeat_back_addr":"172.21.15.79:6806/1823728464","heartbeat_front_addr":"172.21.15.79:6807/1823728464","state":["exists","up"]},{"osd":2,"uuid":"35820c96-5f2a-4e2e-bc4b-714ab8e1c841","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6803","nonce":2585849279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":2585849279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":2585849279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6817","nonce":2585849279}]},"public_addr":"172.21.15.103:6803/2585849279","cluster_addr":"172.21.15.103:6804/2585849279","heartbeat_back_addr":"172.21.15.103:6816/2585849279","heartbeat_front_addr":"172.21.15.103:6817/2585849279","state":["exists","up"]},{"osd":3,"uuid":"da975ea1-eeb6-4619-b73f-786b46dd8fe6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6809","nonce":659350107}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":659350107}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":659350107}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6811","nonce":659350107}]},"public_addr":"172.21.15.103:6809/659350107","cluster_addr":"172.21.15.103:6808/659350107","heartbeat_back_addr":"172.21.15.103:6812/659350107","heartbeat_front_addr":"172.21.15.103:6811/659350107","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":158},{"begin":161,"length":1},{"begin":163,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T02:01:11.303 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-04-24T02:01:11.303 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-04-24T02:01:11.304 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-04-24T02:01:11.304 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-04-24T02:01:11.422 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:01:11.414+0000 7f3dd5aad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:01:11.422 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:01:11.415+0000 7fd34659d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:01:11.423 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:01:11.415+0000 7f769d95d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:01:11.423 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:01:11.415+0000 7f9fecd9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:01:11.425 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:01:11.424+0000 7f769d95d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:01:11.425 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:01:11.424+0000 7f3dd5aad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:01:11.426 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:01:11.425+0000 7f9fecd9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:01:11.426 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:01:11.425+0000 7fd34659d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:01:11.456 INFO:teuthology.orchestra.run.smithi079.stdout:38654706021 2024-04-24T02:01:11.456 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-24T02:01:11.457 INFO:teuthology.orchestra.run.smithi079.stdout:38654706022 2024-04-24T02:01:11.458 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-24T02:01:11.465 INFO:teuthology.orchestra.run.smithi079.stdout:42949673317 2024-04-24T02:01:11.465 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-24T02:01:11.467 INFO:teuthology.orchestra.run.smithi079.stdout:42949673317 2024-04-24T02:01:11.468 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-24T02:01:11.576 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:01:11.570+0000 7f9d3b95d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:01:11.576 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:01:11.570+0000 7fe5d215d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:01:11.580 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:01:11.580+0000 7f9d3b95d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:01:11.581 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:01:11.580+0000 7fe5d215d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:01:11.623 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:01:11.617+0000 7f6fe275d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:01:11.627 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:01:11.626+0000 7f6fe275d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:01:11.634 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:01:11.627+0000 7f5125d5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:01:11.640 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:01:11.639+0000 7f5125d5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:01:11.782 INFO:teuthology.orchestra.run.smithi079.stdout:38654706018 2024-04-24T02:01:11.797 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654706021 got 38654706018 for osd.1 2024-04-24T02:01:11.803 INFO:teuthology.orchestra.run.smithi079.stdout:38654706019 2024-04-24T02:01:11.816 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654706022 got 38654706019 for osd.0 2024-04-24T02:01:11.840 INFO:teuthology.orchestra.run.smithi079.stdout:42949673314 2024-04-24T02:01:11.842 INFO:teuthology.orchestra.run.smithi079.stdout:42949673314 2024-04-24T02:01:11.853 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949673317 got 42949673314 for osd.2 2024-04-24T02:01:11.856 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949673317 got 42949673314 for osd.3 2024-04-24T02:01:12.798 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-24T02:01:12.817 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-24T02:01:12.854 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-24T02:01:12.857 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-24T02:01:12.915 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:01:12.908+0000 7fedc102d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:01:12.919 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:01:12.919+0000 7fedc102d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:01:12.958 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:01:12.952+0000 7ff4c7d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:01:12.963 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:01:12.963+0000 7ff4c7d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:01:13.001 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:01:12.996+0000 7f8aefe6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:01:13.002 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:01:12.996+0000 7faeb055d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:01:13.007 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:01:13.006+0000 7faeb055d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:01:13.008 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:01:13.007+0000 7f8aefe6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:01:13.128 INFO:teuthology.orchestra.run.smithi079.stdout:38654706018 2024-04-24T02:01:13.141 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654706021 got 38654706018 for osd.1 2024-04-24T02:01:13.165 INFO:teuthology.orchestra.run.smithi079.stdout:38654706019 2024-04-24T02:01:13.178 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654706022 got 38654706019 for osd.0 2024-04-24T02:01:13.226 INFO:teuthology.orchestra.run.smithi079.stdout:42949673314 2024-04-24T02:01:13.230 INFO:teuthology.orchestra.run.smithi079.stdout:42949673314 2024-04-24T02:01:13.240 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949673317 got 42949673314 for osd.2 2024-04-24T02:01:13.243 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949673317 got 42949673314 for osd.3 2024-04-24T02:01:14.142 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-24T02:01:14.180 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-24T02:01:14.241 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-24T02:01:14.244 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-24T02:01:14.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:01:14.260+0000 7fa8a5417640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:01:14.272 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:01:14.272+0000 7fa8a5417640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:01:14.310 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:01:14.304+0000 7f218c86d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:01:14.315 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:01:14.315+0000 7f218c86d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:01:14.373 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:01:14.367+0000 7f1176d5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:01:14.377 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:01:14.376+0000 7f1176d5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:01:14.390 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:01:14.382+0000 7f16f699d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:01:14.396 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:01:14.396+0000 7f16f699d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:01:14.531 INFO:teuthology.orchestra.run.smithi079.stdout:38654706021 2024-04-24T02:01:14.544 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654706021 got 38654706021 for osd.1 2024-04-24T02:01:14.544 DEBUG:teuthology.parallel:result is None 2024-04-24T02:01:14.591 INFO:teuthology.orchestra.run.smithi079.stdout:38654706022 2024-04-24T02:01:14.605 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654706022 got 38654706022 for osd.0 2024-04-24T02:01:14.605 INFO:teuthology.orchestra.run.smithi079.stdout:42949673317 2024-04-24T02:01:14.605 INFO:teuthology.orchestra.run.smithi079.stdout:42949673317 2024-04-24T02:01:14.605 DEBUG:teuthology.parallel:result is None 2024-04-24T02:01:14.619 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949673317 got 42949673317 for osd.3 2024-04-24T02:01:14.620 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949673317 got 42949673317 for osd.2 2024-04-24T02:01:14.620 DEBUG:teuthology.parallel:result is None 2024-04-24T02:01:14.620 DEBUG:teuthology.parallel:result is None 2024-04-24T02:01:14.620 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-04-24T02:01:14.620 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T02:01:14.766 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:01:14.762+0000 7fe30f2ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:01:14.770 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:01:14.769+0000 7fe30f2ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:01:14.957 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T02:01:14.958 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-24T02:01:14.970 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":522,"stamp":"2024-04-24T02:01:13.014754+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":94847753,"num_objects":49,"num_object_clones":14,"num_object_copies":147,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":2572,"num_read_kb":1806935,"num_write":2387,"num_write_kb":611215,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4653,"ondisk_log_size":4653,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":353224,"kb_used_data":208488,"kb_used_omap":0,"kb_used_meta":144704,"kb_avail":377134136,"statfs":{"total":386547056640,"available":386185355264,"internally_reserved":0,"allocated":213491712,"data_stored":229204080,"data_compressed":89889,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":148176896},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.358793"},"pg_stats":[{"pgid":"3.f","version":"342'255","reported_seq":1316,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-04-24T02:01:09.679967+0000","last_change":"2024-04-24T02:01:00.252850+0000","last_active":"2024-04-24T02:01:09.679967+0000","last_peered":"2024-04-24T02:01:09.679967+0000","last_clean":"2024-04-24T02:01:09.679967+0000","last_became_active":"2024-04-24T01:55:30.386726+0000","last_became_peered":"2024-04-24T01:55:30.386726+0000","last_unstale":"2024-04-24T02:01:09.679967+0000","last_undegraded":"2024-04-24T02:01:09.679967+0000","last_fullsized":"2024-04-24T02:01:09.679967+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":255,"log_dups_size":0,"ondisk_log_size":255,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":23,"snaptrim_duration":0,"stat_sum":{"num_bytes":5512960,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":185,"num_read_kb":165160,"num_write":116,"num_write_kb":33460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"326'374","reported_seq":1370,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:10.682824+0000","last_change":"2024-04-24T02:01:00.196927+0000","last_active":"2024-04-24T02:01:10.682824+0000","last_peered":"2024-04-24T02:01:10.682824+0000","last_clean":"2024-04-24T02:01:10.682824+0000","last_became_active":"2024-04-24T01:55:30.321866+0000","last_became_peered":"2024-04-24T01:55:30.321866+0000","last_unstale":"2024-04-24T02:01:10.682824+0000","last_undegraded":"2024-04-24T02:01:10.682824+0000","last_fullsized":"2024-04-24T02:01:10.682824+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":374,"log_dups_size":0,"ondisk_log_size":374,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":36,"snaptrim_duration":0,"stat_sum":{"num_bytes":2823708,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":203,"num_read_kb":136089,"num_write":185,"num_write_kb":49191,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"342'235","reported_seq":1248,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-04-24T02:01:10.278967+0000","last_change":"2024-04-24T02:01:00.191968+0000","last_active":"2024-04-24T02:01:10.278967+0000","last_peered":"2024-04-24T02:01:10.278967+0000","last_clean":"2024-04-24T02:01:10.278967+0000","last_became_active":"2024-04-24T01:55:30.320186+0000","last_became_peered":"2024-04-24T01:55:30.320186+0000","last_unstale":"2024-04-24T02:01:10.278967+0000","last_undegraded":"2024-04-24T02:01:10.278967+0000","last_fullsized":"2024-04-24T02:01:10.278967+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":235,"log_dups_size":0,"ondisk_log_size":235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":24,"snaptrim_duration":0,"stat_sum":{"num_bytes":6017104,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":139,"num_read_kb":94253,"num_write":128,"num_write_kb":35473,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"342'309","reported_seq":1344,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-04-24T02:01:09.678868+0000","last_change":"2024-04-24T02:01:00.248214+0000","last_active":"2024-04-24T02:01:09.678868+0000","last_peered":"2024-04-24T02:01:09.678868+0000","last_clean":"2024-04-24T02:01:09.678868+0000","last_became_active":"2024-04-24T01:55:30.288081+0000","last_became_peered":"2024-04-24T01:55:30.288081+0000","last_unstale":"2024-04-24T02:01:09.678868+0000","last_undegraded":"2024-04-24T02:01:09.678868+0000","last_fullsized":"2024-04-24T02:01:09.678868+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":309,"log_dups_size":0,"ondisk_log_size":309,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":30,"snaptrim_duration":0,"stat_sum":{"num_bytes":9832458,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":161,"num_read_kb":116003,"num_write":154,"num_write_kb":41482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"336'138","reported_seq":1182,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-04-24T02:01:10.314365+0000","last_change":"2024-04-24T02:00:59.336871+0000","last_active":"2024-04-24T02:01:10.314365+0000","last_peered":"2024-04-24T02:01:10.314365+0000","last_clean":"2024-04-24T02:01:10.314365+0000","last_became_active":"2024-04-24T01:55:30.333815+0000","last_became_peered":"2024-04-24T01:55:30.333815+0000","last_unstale":"2024-04-24T02:01:10.314365+0000","last_undegraded":"2024-04-24T02:01:10.314365+0000","last_fullsized":"2024-04-24T02:01:10.314365+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":17,"snaptrim_duration":0,"stat_sum":{"num_bytes":2857322,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":82,"num_read_kb":52358,"num_write":76,"num_write_kb":20051,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"342'638","reported_seq":1605,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-04-24T02:01:09.677697+0000","last_change":"2024-04-24T02:01:00.250311+0000","last_active":"2024-04-24T02:01:09.677697+0000","last_peered":"2024-04-24T02:01:09.677697+0000","last_clean":"2024-04-24T02:01:09.677697+0000","last_became_active":"2024-04-24T01:55:30.315075+0000","last_became_peered":"2024-04-24T01:55:30.315075+0000","last_unstale":"2024-04-24T02:01:09.677697+0000","last_undegraded":"2024-04-24T02:01:09.677697+0000","last_fullsized":"2024-04-24T02:01:09.677697+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":638,"log_dups_size":0,"ondisk_log_size":638,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":65,"snaptrim_duration":0,"stat_sum":{"num_bytes":8807264,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":282,"num_read_kb":182263,"num_write":282,"num_write_kb":72883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"342'150","reported_seq":1195,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-04-24T02:01:09.684332+0000","last_change":"2024-04-24T02:01:00.239736+0000","last_active":"2024-04-24T02:01:09.684332+0000","last_peered":"2024-04-24T02:01:09.684332+0000","last_clean":"2024-04-24T02:01:09.684332+0000","last_became_active":"2024-04-24T01:55:30.350300+0000","last_became_peered":"2024-04-24T01:55:30.350300+0000","last_unstale":"2024-04-24T02:01:09.684332+0000","last_undegraded":"2024-04-24T02:01:09.684332+0000","last_fullsized":"2024-04-24T02:01:09.684332+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":150,"log_dups_size":0,"ondisk_log_size":150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":13,"snaptrim_duration":0,"stat_sum":{"num_bytes":3243883,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":66,"num_read_kb":40460,"num_write":87,"num_write_kb":22401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"264'34","reported_seq":1056,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-04-24T02:01:10.293253+0000","last_change":"2024-04-24T02:01:00.170396+0000","last_active":"2024-04-24T02:01:10.293253+0000","last_peered":"2024-04-24T02:01:10.293253+0000","last_clean":"2024-04-24T02:01:10.293253+0000","last_became_active":"2024-04-24T01:55:30.376813+0000","last_became_peered":"2024-04-24T01:55:30.376813+0000","last_unstale":"2024-04-24T02:01:10.293253+0000","last_undegraded":"2024-04-24T02:01:10.293253+0000","last_fullsized":"2024-04-24T02:01:10.293253+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0,"stat_sum":{"num_bytes":3025359,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":21,"num_read_kb":15411,"num_write":27,"num_write_kb":6802,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'97","reported_seq":1028,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-04-24T02:01:10.299132+0000","last_change":"2024-04-24T01:55:16.196470+0000","last_active":"2024-04-24T02:01:10.299132+0000","last_peered":"2024-04-24T02:01:10.299132+0000","last_clean":"2024-04-24T02:01:10.299132+0000","last_became_active":"2024-04-24T01:55:16.190464+0000","last_became_peered":"2024-04-24T01:55:16.190464+0000","last_unstale":"2024-04-24T02:01:10.299132+0000","last_undegraded":"2024-04-24T02:01:10.299132+0000","last_fullsized":"2024-04-24T02:01:10.299132+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:15.127115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:15.127115+0000","last_clean_scrub_stamp":"2024-04-24T01:55:15.127115+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":869,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-04-24T02:01:10.293101+0000","last_change":"2024-04-24T01:55:18.236148+0000","last_active":"2024-04-24T02:01:10.293101+0000","last_peered":"2024-04-24T02:01:10.293101+0000","last_clean":"2024-04-24T02:01:10.293101+0000","last_became_active":"2024-04-24T01:55:18.230154+0000","last_became_peered":"2024-04-24T01:55:18.230154+0000","last_unstale":"2024-04-24T02:01:10.293101+0000","last_undegraded":"2024-04-24T02:01:10.293101+0000","last_fullsized":"2024-04-24T02:01:10.293101+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"342'462","reported_seq":1455,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-04-24T02:01:10.292145+0000","last_change":"2024-04-24T02:01:00.337592+0000","last_active":"2024-04-24T02:01:10.292145+0000","last_peered":"2024-04-24T02:01:10.292145+0000","last_clean":"2024-04-24T02:01:10.292145+0000","last_became_active":"2024-04-24T01:55:30.316222+0000","last_became_peered":"2024-04-24T01:55:30.316222+0000","last_unstale":"2024-04-24T02:01:10.292145+0000","last_undegraded":"2024-04-24T02:01:10.292145+0000","last_fullsized":"2024-04-24T02:01:10.292145+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":462,"log_dups_size":0,"ondisk_log_size":462,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":39,"snaptrim_duration":0,"stat_sum":{"num_bytes":3025392,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":231,"num_read_kb":163233,"num_write":236,"num_write_kb":65892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":905,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-04-24T02:01:09.686630+0000","last_change":"2024-04-24T01:55:18.265905+0000","last_active":"2024-04-24T02:01:09.686630+0000","last_peered":"2024-04-24T02:01:09.686630+0000","last_clean":"2024-04-24T02:01:09.686630+0000","last_became_active":"2024-04-24T01:55:18.257020+0000","last_became_peered":"2024-04-24T01:55:18.257020+0000","last_unstale":"2024-04-24T02:01:09.686630+0000","last_undegraded":"2024-04-24T02:01:09.686630+0000","last_fullsized":"2024-04-24T02:01:09.686630+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"342'358","reported_seq":1364,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-04-24T02:01:10.301289+0000","last_change":"2024-04-24T02:00:59.359790+0000","last_active":"2024-04-24T02:01:10.301289+0000","last_peered":"2024-04-24T02:01:10.301289+0000","last_clean":"2024-04-24T02:01:10.301289+0000","last_became_active":"2024-04-24T01:55:30.344129+0000","last_became_peered":"2024-04-24T01:55:30.344129+0000","last_unstale":"2024-04-24T02:01:10.301289+0000","last_undegraded":"2024-04-24T02:01:10.301289+0000","last_fullsized":"2024-04-24T02:01:10.301289+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":358,"log_dups_size":0,"ondisk_log_size":358,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":31,"snaptrim_duration":0,"stat_sum":{"num_bytes":10420703,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":155,"num_read_kb":92133,"num_write":178,"num_write_kb":50302,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":869,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-04-24T02:01:10.277999+0000","last_change":"2024-04-24T01:55:18.215251+0000","last_active":"2024-04-24T02:01:10.277999+0000","last_peered":"2024-04-24T02:01:10.277999+0000","last_clean":"2024-04-24T02:01:10.277999+0000","last_became_active":"2024-04-24T01:55:18.206605+0000","last_became_peered":"2024-04-24T01:55:18.206605+0000","last_unstale":"2024-04-24T02:01:10.277999+0000","last_undegraded":"2024-04-24T02:01:10.277999+0000","last_fullsized":"2024-04-24T02:01:10.277999+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"342'208","reported_seq":1256,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-04-24T02:01:10.312146+0000","last_change":"2024-04-24T02:00:59.354665+0000","last_active":"2024-04-24T02:01:10.312146+0000","last_peered":"2024-04-24T02:01:10.312146+0000","last_clean":"2024-04-24T02:01:10.312146+0000","last_became_active":"2024-04-24T01:55:30.401366+0000","last_became_peered":"2024-04-24T01:55:30.401366+0000","last_unstale":"2024-04-24T02:01:10.312146+0000","last_undegraded":"2024-04-24T02:01:10.312146+0000","last_fullsized":"2024-04-24T02:01:10.312146+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":208,"log_dups_size":0,"ondisk_log_size":208,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":18,"snaptrim_duration":0,"stat_sum":{"num_bytes":3327918,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":153,"num_read_kb":115484,"num_write":85,"num_write_kb":21155,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":905,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-04-24T02:01:09.685694+0000","last_change":"2024-04-24T01:55:18.230893+0000","last_active":"2024-04-24T02:01:09.685694+0000","last_peered":"2024-04-24T02:01:09.685694+0000","last_clean":"2024-04-24T02:01:09.685694+0000","last_became_active":"2024-04-24T01:55:18.225188+0000","last_became_peered":"2024-04-24T01:55:18.225188+0000","last_unstale":"2024-04-24T02:01:09.685694+0000","last_undegraded":"2024-04-24T02:01:09.685694+0000","last_fullsized":"2024-04-24T02:01:09.685694+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"342'294","reported_seq":1287,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:10.678999+0000","last_change":"2024-04-24T02:01:00.246736+0000","last_active":"2024-04-24T02:01:10.678999+0000","last_peered":"2024-04-24T02:01:10.678999+0000","last_clean":"2024-04-24T02:01:10.678999+0000","last_became_active":"2024-04-24T01:55:30.343318+0000","last_became_peered":"2024-04-24T01:55:30.343318+0000","last_unstale":"2024-04-24T02:01:10.678999+0000","last_undegraded":"2024-04-24T02:01:10.678999+0000","last_fullsized":"2024-04-24T02:01:10.678999+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":294,"log_dups_size":0,"ondisk_log_size":294,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":26,"snaptrim_duration":0,"stat_sum":{"num_bytes":10118177,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":153,"num_read_kb":122486,"num_write":138,"num_write_kb":38234,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"342'437","reported_seq":1431,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-04-24T02:01:10.291082+0000","last_change":"2024-04-24T02:01:00.338179+0000","last_active":"2024-04-24T02:01:10.291082+0000","last_peered":"2024-04-24T02:01:10.291082+0000","last_clean":"2024-04-24T02:01:10.291082+0000","last_became_active":"2024-04-24T01:55:30.345308+0000","last_became_peered":"2024-04-24T01:55:30.345308+0000","last_unstale":"2024-04-24T02:01:10.291082+0000","last_undegraded":"2024-04-24T02:01:10.291082+0000","last_fullsized":"2024-04-24T02:01:10.291082+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":437,"log_dups_size":0,"ondisk_log_size":437,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":37,"snaptrim_duration":0,"stat_sum":{"num_bytes":10050949,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":237,"num_read_kb":172356,"num_write":211,"num_write_kb":56847,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":866,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:10.688889+0000","last_change":"2024-04-24T01:55:18.224238+0000","last_active":"2024-04-24T02:01:10.688889+0000","last_peered":"2024-04-24T02:01:10.688889+0000","last_clean":"2024-04-24T02:01:10.688889+0000","last_became_active":"2024-04-24T01:55:18.219131+0000","last_became_peered":"2024-04-24T01:55:18.219131+0000","last_unstale":"2024-04-24T02:01:10.688889+0000","last_undegraded":"2024-04-24T02:01:10.688889+0000","last_fullsized":"2024-04-24T02:01:10.688889+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"342'103","reported_seq":1116,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:10.693419+0000","last_change":"2024-04-24T02:01:00.249384+0000","last_active":"2024-04-24T02:01:10.693419+0000","last_peered":"2024-04-24T02:01:10.693419+0000","last_clean":"2024-04-24T02:01:10.693419+0000","last_became_active":"2024-04-24T01:55:30.319958+0000","last_became_peered":"2024-04-24T01:55:30.319958+0000","last_unstale":"2024-04-24T02:01:10.693419+0000","last_undegraded":"2024-04-24T02:01:10.693419+0000","last_fullsized":"2024-04-24T02:01:10.693419+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":8,"snaptrim_duration":0,"stat_sum":{"num_bytes":3445567,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":50,"num_read_kb":44680,"num_write":61,"num_write_kb":17461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":895,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-04-24T02:01:10.312495+0000","last_change":"2024-04-24T01:55:18.235763+0000","last_active":"2024-04-24T02:01:10.312495+0000","last_peered":"2024-04-24T02:01:10.312495+0000","last_clean":"2024-04-24T02:01:10.312495+0000","last_became_active":"2024-04-24T01:55:18.229175+0000","last_became_peered":"2024-04-24T01:55:18.229175+0000","last_unstale":"2024-04-24T02:01:10.312495+0000","last_undegraded":"2024-04-24T02:01:10.312495+0000","last_fullsized":"2024-04-24T02:01:10.312495+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"336'192","reported_seq":1225,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-04-24T02:01:10.304579+0000","last_change":"2024-04-24T02:00:59.343900+0000","last_active":"2024-04-24T02:01:10.304579+0000","last_peered":"2024-04-24T02:01:10.304579+0000","last_clean":"2024-04-24T02:01:10.304579+0000","last_became_active":"2024-04-24T01:55:30.372690+0000","last_became_peered":"2024-04-24T01:55:30.372690+0000","last_unstale":"2024-04-24T02:01:10.304579+0000","last_undegraded":"2024-04-24T02:01:10.304579+0000","last_fullsized":"2024-04-24T02:01:10.304579+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":192,"log_dups_size":0,"ondisk_log_size":192,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":18,"snaptrim_duration":0,"stat_sum":{"num_bytes":5563381,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":102,"num_read_kb":75985,"num_write":95,"num_write_kb":27048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":866,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:10.694645+0000","last_change":"2024-04-24T01:55:18.278351+0000","last_active":"2024-04-24T02:01:10.694645+0000","last_peered":"2024-04-24T02:01:10.694645+0000","last_clean":"2024-04-24T02:01:10.694645+0000","last_became_active":"2024-04-24T01:55:18.273668+0000","last_became_peered":"2024-04-24T01:55:18.273668+0000","last_unstale":"2024-04-24T02:01:10.694645+0000","last_undegraded":"2024-04-24T02:01:10.694645+0000","last_fullsized":"2024-04-24T02:01:10.694645+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"342'369","reported_seq":1391,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:10.689816+0000","last_change":"2024-04-24T02:01:00.639502+0000","last_active":"2024-04-24T02:01:10.689816+0000","last_peered":"2024-04-24T02:01:10.689816+0000","last_clean":"2024-04-24T02:01:10.689816+0000","last_became_active":"2024-04-24T01:55:30.363123+0000","last_became_peered":"2024-04-24T01:55:30.363123+0000","last_unstale":"2024-04-24T02:01:10.689816+0000","last_undegraded":"2024-04-24T02:01:10.689816+0000","last_fullsized":"2024-04-24T02:01:10.689816+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":369,"log_dups_size":0,"ondisk_log_size":369,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":41,"snaptrim_duration":0,"stat_sum":{"num_bytes":6185240,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":249,"num_read_kb":218490,"num_write":191,"num_write_kb":50639,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":869,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-04-24T02:01:10.290236+0000","last_change":"2024-04-24T01:55:18.241117+0000","last_active":"2024-04-24T02:01:10.290236+0000","last_peered":"2024-04-24T02:01:10.290236+0000","last_clean":"2024-04-24T02:01:10.290236+0000","last_became_active":"2024-04-24T01:55:18.234313+0000","last_became_peered":"2024-04-24T01:55:18.234313+0000","last_unstale":"2024-04-24T02:01:10.290236+0000","last_undegraded":"2024-04-24T02:01:10.290236+0000","last_fullsized":"2024-04-24T02:01:10.290236+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":94257385,"num_objects":47,"num_object_clones":14,"num_object_copies":141,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":2469,"num_read_kb":1806844,"num_write":2250,"num_write_kb":609321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":211218432,"data_stored":226516047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4556,"ondisk_log_size":4556,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":1}],"osd_stats":[{"osd":3,"up_from":10,"seq":42949673317,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":99692,"kb_used_data":60388,"kb_used_omap":0,"kb_used_meta":39296,"kb_avail":94272148,"statfs":{"total":96636764160,"available":96534679552,"internally_reserved":0,"allocated":61837312,"data_stored":66192113,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":40239104},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673317,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":82876,"kb_used_data":47732,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94288964,"statfs":{"total":96636764160,"available":96551899136,"internally_reserved":0,"allocated":48877568,"data_stored":53464844,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706022,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":82876,"kb_used_data":47732,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94288964,"statfs":{"total":96636764160,"available":96551899136,"internally_reserved":0,"allocated":48877568,"data_stored":52344808,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706021,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":87780,"kb_used_data":52636,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94284060,"statfs":{"total":96636764160,"available":96546877440,"internally_reserved":0,"allocated":53899264,"data_stored":57202315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":48295936,"data_stored":51526038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53379072,"data_stored":56973913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":48287744,"data_stored":52642753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61255680,"data_stored":65373343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T02:01:14.972 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T02:01:15.081 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:01:15.076+0000 7f1826a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:01:15.084 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:01:15.084+0000 7f1826a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:01:15.271 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T02:01:15.272 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-24T02:01:15.284 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":523,"stamp":"2024-04-24T02:01:15.015551+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":94847753,"num_objects":49,"num_object_clones":14,"num_object_copies":147,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":2572,"num_read_kb":1806935,"num_write":2387,"num_write_kb":611215,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4653,"ondisk_log_size":4653,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":353248,"kb_used_data":208512,"kb_used_omap":0,"kb_used_meta":144704,"kb_avail":377134112,"statfs":{"total":386547056640,"available":386185330688,"internally_reserved":0,"allocated":213516288,"data_stored":229214043,"data_compressed":89889,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":148176896},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.358679"},"pg_stats":[{"pgid":"3.f","version":"342'255","reported_seq":1319,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:12.191911+0000","last_change":"2024-04-24T02:01:00.252850+0000","last_active":"2024-04-24T02:01:12.191911+0000","last_peered":"2024-04-24T02:01:12.191911+0000","last_clean":"2024-04-24T02:01:12.191911+0000","last_became_active":"2024-04-24T01:55:30.386726+0000","last_became_peered":"2024-04-24T01:55:30.386726+0000","last_unstale":"2024-04-24T02:01:12.191911+0000","last_undegraded":"2024-04-24T02:01:12.191911+0000","last_fullsized":"2024-04-24T02:01:12.191911+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":255,"log_dups_size":0,"ondisk_log_size":255,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":23,"snaptrim_duration":0,"stat_sum":{"num_bytes":5512960,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":185,"num_read_kb":165160,"num_write":116,"num_write_kb":33460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"326'374","reported_seq":1370,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:10.682824+0000","last_change":"2024-04-24T02:01:00.196927+0000","last_active":"2024-04-24T02:01:10.682824+0000","last_peered":"2024-04-24T02:01:10.682824+0000","last_clean":"2024-04-24T02:01:10.682824+0000","last_became_active":"2024-04-24T01:55:30.321866+0000","last_became_peered":"2024-04-24T01:55:30.321866+0000","last_unstale":"2024-04-24T02:01:10.682824+0000","last_undegraded":"2024-04-24T02:01:10.682824+0000","last_fullsized":"2024-04-24T02:01:10.682824+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":374,"log_dups_size":0,"ondisk_log_size":374,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":36,"snaptrim_duration":0,"stat_sum":{"num_bytes":2823708,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":203,"num_read_kb":136089,"num_write":185,"num_write_kb":49191,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"342'235","reported_seq":1251,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:12.147307+0000","last_change":"2024-04-24T02:01:00.191968+0000","last_active":"2024-04-24T02:01:12.147307+0000","last_peered":"2024-04-24T02:01:12.147307+0000","last_clean":"2024-04-24T02:01:12.147307+0000","last_became_active":"2024-04-24T01:55:30.320186+0000","last_became_peered":"2024-04-24T01:55:30.320186+0000","last_unstale":"2024-04-24T02:01:12.147307+0000","last_undegraded":"2024-04-24T02:01:12.147307+0000","last_fullsized":"2024-04-24T02:01:12.147307+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":235,"log_dups_size":0,"ondisk_log_size":235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":24,"snaptrim_duration":0,"stat_sum":{"num_bytes":6017104,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":139,"num_read_kb":94253,"num_write":128,"num_write_kb":35473,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"342'309","reported_seq":1347,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:12.192818+0000","last_change":"2024-04-24T02:01:00.248214+0000","last_active":"2024-04-24T02:01:12.192818+0000","last_peered":"2024-04-24T02:01:12.192818+0000","last_clean":"2024-04-24T02:01:12.192818+0000","last_became_active":"2024-04-24T01:55:30.288081+0000","last_became_peered":"2024-04-24T01:55:30.288081+0000","last_unstale":"2024-04-24T02:01:12.192818+0000","last_undegraded":"2024-04-24T02:01:12.192818+0000","last_fullsized":"2024-04-24T02:01:12.192818+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":309,"log_dups_size":0,"ondisk_log_size":309,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":30,"snaptrim_duration":0,"stat_sum":{"num_bytes":9832458,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":161,"num_read_kb":116003,"num_write":154,"num_write_kb":41482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"336'138","reported_seq":1185,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:13.184844+0000","last_change":"2024-04-24T02:00:59.336871+0000","last_active":"2024-04-24T02:01:13.184844+0000","last_peered":"2024-04-24T02:01:13.184844+0000","last_clean":"2024-04-24T02:01:13.184844+0000","last_became_active":"2024-04-24T01:55:30.333815+0000","last_became_peered":"2024-04-24T01:55:30.333815+0000","last_unstale":"2024-04-24T02:01:13.184844+0000","last_undegraded":"2024-04-24T02:01:13.184844+0000","last_fullsized":"2024-04-24T02:01:13.184844+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":17,"snaptrim_duration":0,"stat_sum":{"num_bytes":2857322,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":82,"num_read_kb":52358,"num_write":76,"num_write_kb":20051,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"342'638","reported_seq":1608,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:12.193652+0000","last_change":"2024-04-24T02:01:00.250311+0000","last_active":"2024-04-24T02:01:12.193652+0000","last_peered":"2024-04-24T02:01:12.193652+0000","last_clean":"2024-04-24T02:01:12.193652+0000","last_became_active":"2024-04-24T01:55:30.315075+0000","last_became_peered":"2024-04-24T01:55:30.315075+0000","last_unstale":"2024-04-24T02:01:12.193652+0000","last_undegraded":"2024-04-24T02:01:12.193652+0000","last_fullsized":"2024-04-24T02:01:12.193652+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":638,"log_dups_size":0,"ondisk_log_size":638,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":65,"snaptrim_duration":0,"stat_sum":{"num_bytes":8807264,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":282,"num_read_kb":182263,"num_write":282,"num_write_kb":72883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"342'150","reported_seq":1198,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:12.200097+0000","last_change":"2024-04-24T02:01:00.239736+0000","last_active":"2024-04-24T02:01:12.200097+0000","last_peered":"2024-04-24T02:01:12.200097+0000","last_clean":"2024-04-24T02:01:12.200097+0000","last_became_active":"2024-04-24T01:55:30.350300+0000","last_became_peered":"2024-04-24T01:55:30.350300+0000","last_unstale":"2024-04-24T02:01:12.200097+0000","last_undegraded":"2024-04-24T02:01:12.200097+0000","last_fullsized":"2024-04-24T02:01:12.200097+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":150,"log_dups_size":0,"ondisk_log_size":150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":13,"snaptrim_duration":0,"stat_sum":{"num_bytes":3243883,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":66,"num_read_kb":40460,"num_write":87,"num_write_kb":22401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"264'34","reported_seq":1059,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:12.155738+0000","last_change":"2024-04-24T02:01:00.170396+0000","last_active":"2024-04-24T02:01:12.155738+0000","last_peered":"2024-04-24T02:01:12.155738+0000","last_clean":"2024-04-24T02:01:12.155738+0000","last_became_active":"2024-04-24T01:55:30.376813+0000","last_became_peered":"2024-04-24T01:55:30.376813+0000","last_unstale":"2024-04-24T02:01:12.155738+0000","last_undegraded":"2024-04-24T02:01:12.155738+0000","last_fullsized":"2024-04-24T02:01:12.155738+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0,"stat_sum":{"num_bytes":3025359,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":21,"num_read_kb":15411,"num_write":27,"num_write_kb":6802,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'97","reported_seq":1031,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:13.169480+0000","last_change":"2024-04-24T01:55:16.196470+0000","last_active":"2024-04-24T02:01:13.169480+0000","last_peered":"2024-04-24T02:01:13.169480+0000","last_clean":"2024-04-24T02:01:13.169480+0000","last_became_active":"2024-04-24T01:55:16.190464+0000","last_became_peered":"2024-04-24T01:55:16.190464+0000","last_unstale":"2024-04-24T02:01:13.169480+0000","last_undegraded":"2024-04-24T02:01:13.169480+0000","last_fullsized":"2024-04-24T02:01:13.169480+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:15.127115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:15.127115+0000","last_clean_scrub_stamp":"2024-04-24T01:55:15.127115+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":872,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:12.153789+0000","last_change":"2024-04-24T01:55:18.236148+0000","last_active":"2024-04-24T02:01:12.153789+0000","last_peered":"2024-04-24T02:01:12.153789+0000","last_clean":"2024-04-24T02:01:12.153789+0000","last_became_active":"2024-04-24T01:55:18.230154+0000","last_became_peered":"2024-04-24T01:55:18.230154+0000","last_unstale":"2024-04-24T02:01:12.153789+0000","last_undegraded":"2024-04-24T02:01:12.153789+0000","last_fullsized":"2024-04-24T02:01:12.153789+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"342'462","reported_seq":1458,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:12.153249+0000","last_change":"2024-04-24T02:01:00.337592+0000","last_active":"2024-04-24T02:01:12.153249+0000","last_peered":"2024-04-24T02:01:12.153249+0000","last_clean":"2024-04-24T02:01:12.153249+0000","last_became_active":"2024-04-24T01:55:30.316222+0000","last_became_peered":"2024-04-24T01:55:30.316222+0000","last_unstale":"2024-04-24T02:01:12.153249+0000","last_undegraded":"2024-04-24T02:01:12.153249+0000","last_fullsized":"2024-04-24T02:01:12.153249+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":462,"log_dups_size":0,"ondisk_log_size":462,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":39,"snaptrim_duration":0,"stat_sum":{"num_bytes":3025392,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":231,"num_read_kb":163233,"num_write":236,"num_write_kb":65892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":908,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:12.201364+0000","last_change":"2024-04-24T01:55:18.265905+0000","last_active":"2024-04-24T02:01:12.201364+0000","last_peered":"2024-04-24T02:01:12.201364+0000","last_clean":"2024-04-24T02:01:12.201364+0000","last_became_active":"2024-04-24T01:55:18.257020+0000","last_became_peered":"2024-04-24T01:55:18.257020+0000","last_unstale":"2024-04-24T02:01:12.201364+0000","last_undegraded":"2024-04-24T02:01:12.201364+0000","last_fullsized":"2024-04-24T02:01:12.201364+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"342'358","reported_seq":1367,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:13.173644+0000","last_change":"2024-04-24T02:00:59.359790+0000","last_active":"2024-04-24T02:01:13.173644+0000","last_peered":"2024-04-24T02:01:13.173644+0000","last_clean":"2024-04-24T02:01:13.173644+0000","last_became_active":"2024-04-24T01:55:30.344129+0000","last_became_peered":"2024-04-24T01:55:30.344129+0000","last_unstale":"2024-04-24T02:01:13.173644+0000","last_undegraded":"2024-04-24T02:01:13.173644+0000","last_fullsized":"2024-04-24T02:01:13.173644+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":358,"log_dups_size":0,"ondisk_log_size":358,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":31,"snaptrim_duration":0,"stat_sum":{"num_bytes":10420703,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":155,"num_read_kb":92133,"num_write":178,"num_write_kb":50302,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":872,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:12.146036+0000","last_change":"2024-04-24T01:55:18.215251+0000","last_active":"2024-04-24T02:01:12.146036+0000","last_peered":"2024-04-24T02:01:12.146036+0000","last_clean":"2024-04-24T02:01:12.146036+0000","last_became_active":"2024-04-24T01:55:18.206605+0000","last_became_peered":"2024-04-24T01:55:18.206605+0000","last_unstale":"2024-04-24T02:01:12.146036+0000","last_undegraded":"2024-04-24T02:01:12.146036+0000","last_fullsized":"2024-04-24T02:01:12.146036+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"342'208","reported_seq":1259,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:13.189650+0000","last_change":"2024-04-24T02:00:59.354665+0000","last_active":"2024-04-24T02:01:13.189650+0000","last_peered":"2024-04-24T02:01:13.189650+0000","last_clean":"2024-04-24T02:01:13.189650+0000","last_became_active":"2024-04-24T01:55:30.401366+0000","last_became_peered":"2024-04-24T01:55:30.401366+0000","last_unstale":"2024-04-24T02:01:13.189650+0000","last_undegraded":"2024-04-24T02:01:13.189650+0000","last_fullsized":"2024-04-24T02:01:13.189650+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":208,"log_dups_size":0,"ondisk_log_size":208,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":18,"snaptrim_duration":0,"stat_sum":{"num_bytes":3327918,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":153,"num_read_kb":115484,"num_write":85,"num_write_kb":21155,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":908,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:12.202080+0000","last_change":"2024-04-24T01:55:18.230893+0000","last_active":"2024-04-24T02:01:12.202080+0000","last_peered":"2024-04-24T02:01:12.202080+0000","last_clean":"2024-04-24T02:01:12.202080+0000","last_became_active":"2024-04-24T01:55:18.225188+0000","last_became_peered":"2024-04-24T01:55:18.225188+0000","last_unstale":"2024-04-24T02:01:12.202080+0000","last_undegraded":"2024-04-24T02:01:12.202080+0000","last_fullsized":"2024-04-24T02:01:12.202080+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"342'294","reported_seq":1287,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:10.678999+0000","last_change":"2024-04-24T02:01:00.246736+0000","last_active":"2024-04-24T02:01:10.678999+0000","last_peered":"2024-04-24T02:01:10.678999+0000","last_clean":"2024-04-24T02:01:10.678999+0000","last_became_active":"2024-04-24T01:55:30.343318+0000","last_became_peered":"2024-04-24T01:55:30.343318+0000","last_unstale":"2024-04-24T02:01:10.678999+0000","last_undegraded":"2024-04-24T02:01:10.678999+0000","last_fullsized":"2024-04-24T02:01:10.678999+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":294,"log_dups_size":0,"ondisk_log_size":294,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":26,"snaptrim_duration":0,"stat_sum":{"num_bytes":10118177,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":153,"num_read_kb":122486,"num_write":138,"num_write_kb":38234,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"342'437","reported_seq":1434,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:12.157896+0000","last_change":"2024-04-24T02:01:00.338179+0000","last_active":"2024-04-24T02:01:12.157896+0000","last_peered":"2024-04-24T02:01:12.157896+0000","last_clean":"2024-04-24T02:01:12.157896+0000","last_became_active":"2024-04-24T01:55:30.345308+0000","last_became_peered":"2024-04-24T01:55:30.345308+0000","last_unstale":"2024-04-24T02:01:12.157896+0000","last_undegraded":"2024-04-24T02:01:12.157896+0000","last_fullsized":"2024-04-24T02:01:12.157896+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":437,"log_dups_size":0,"ondisk_log_size":437,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":37,"snaptrim_duration":0,"stat_sum":{"num_bytes":10050949,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":237,"num_read_kb":172356,"num_write":211,"num_write_kb":56847,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":866,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:10.688889+0000","last_change":"2024-04-24T01:55:18.224238+0000","last_active":"2024-04-24T02:01:10.688889+0000","last_peered":"2024-04-24T02:01:10.688889+0000","last_clean":"2024-04-24T02:01:10.688889+0000","last_became_active":"2024-04-24T01:55:18.219131+0000","last_became_peered":"2024-04-24T01:55:18.219131+0000","last_unstale":"2024-04-24T02:01:10.688889+0000","last_undegraded":"2024-04-24T02:01:10.688889+0000","last_fullsized":"2024-04-24T02:01:10.688889+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"342'103","reported_seq":1116,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:10.693419+0000","last_change":"2024-04-24T02:01:00.249384+0000","last_active":"2024-04-24T02:01:10.693419+0000","last_peered":"2024-04-24T02:01:10.693419+0000","last_clean":"2024-04-24T02:01:10.693419+0000","last_became_active":"2024-04-24T01:55:30.319958+0000","last_became_peered":"2024-04-24T01:55:30.319958+0000","last_unstale":"2024-04-24T02:01:10.693419+0000","last_undegraded":"2024-04-24T02:01:10.693419+0000","last_fullsized":"2024-04-24T02:01:10.693419+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":8,"snaptrim_duration":0,"stat_sum":{"num_bytes":3445567,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":50,"num_read_kb":44680,"num_write":61,"num_write_kb":17461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":898,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:13.179399+0000","last_change":"2024-04-24T01:55:18.235763+0000","last_active":"2024-04-24T02:01:13.179399+0000","last_peered":"2024-04-24T02:01:13.179399+0000","last_clean":"2024-04-24T02:01:13.179399+0000","last_became_active":"2024-04-24T01:55:18.229175+0000","last_became_peered":"2024-04-24T01:55:18.229175+0000","last_unstale":"2024-04-24T02:01:13.179399+0000","last_undegraded":"2024-04-24T02:01:13.179399+0000","last_fullsized":"2024-04-24T02:01:13.179399+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"336'192","reported_seq":1228,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:13.172461+0000","last_change":"2024-04-24T02:00:59.343900+0000","last_active":"2024-04-24T02:01:13.172461+0000","last_peered":"2024-04-24T02:01:13.172461+0000","last_clean":"2024-04-24T02:01:13.172461+0000","last_became_active":"2024-04-24T01:55:30.372690+0000","last_became_peered":"2024-04-24T01:55:30.372690+0000","last_unstale":"2024-04-24T02:01:13.172461+0000","last_undegraded":"2024-04-24T02:01:13.172461+0000","last_fullsized":"2024-04-24T02:01:13.172461+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":192,"log_dups_size":0,"ondisk_log_size":192,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":18,"snaptrim_duration":0,"stat_sum":{"num_bytes":5563381,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":102,"num_read_kb":75985,"num_write":95,"num_write_kb":27048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":866,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:10.694645+0000","last_change":"2024-04-24T01:55:18.278351+0000","last_active":"2024-04-24T02:01:10.694645+0000","last_peered":"2024-04-24T02:01:10.694645+0000","last_clean":"2024-04-24T02:01:10.694645+0000","last_became_active":"2024-04-24T01:55:18.273668+0000","last_became_peered":"2024-04-24T01:55:18.273668+0000","last_unstale":"2024-04-24T02:01:10.694645+0000","last_undegraded":"2024-04-24T02:01:10.694645+0000","last_fullsized":"2024-04-24T02:01:10.694645+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"342'369","reported_seq":1391,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:10.689816+0000","last_change":"2024-04-24T02:01:00.639502+0000","last_active":"2024-04-24T02:01:10.689816+0000","last_peered":"2024-04-24T02:01:10.689816+0000","last_clean":"2024-04-24T02:01:10.689816+0000","last_became_active":"2024-04-24T01:55:30.363123+0000","last_became_peered":"2024-04-24T01:55:30.363123+0000","last_unstale":"2024-04-24T02:01:10.689816+0000","last_undegraded":"2024-04-24T02:01:10.689816+0000","last_fullsized":"2024-04-24T02:01:10.689816+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":369,"log_dups_size":0,"ondisk_log_size":369,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":41,"snaptrim_duration":0,"stat_sum":{"num_bytes":6185240,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":249,"num_read_kb":218490,"num_write":191,"num_write_kb":50639,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":872,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:12.156807+0000","last_change":"2024-04-24T01:55:18.241117+0000","last_active":"2024-04-24T02:01:12.156807+0000","last_peered":"2024-04-24T02:01:12.156807+0000","last_clean":"2024-04-24T02:01:12.156807+0000","last_became_active":"2024-04-24T01:55:18.234313+0000","last_became_peered":"2024-04-24T01:55:18.234313+0000","last_unstale":"2024-04-24T02:01:12.156807+0000","last_undegraded":"2024-04-24T02:01:12.156807+0000","last_fullsized":"2024-04-24T02:01:12.156807+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":94257385,"num_objects":47,"num_object_clones":14,"num_object_copies":141,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":2469,"num_read_kb":1806844,"num_write":2250,"num_write_kb":609321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":211218432,"data_stored":226516047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4556,"ondisk_log_size":4556,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":1}],"osd_stats":[{"osd":3,"up_from":10,"seq":42949673319,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":99700,"kb_used_data":60396,"kb_used_omap":0,"kb_used_meta":39296,"kb_avail":94272140,"statfs":{"total":96636764160,"available":96534671360,"internally_reserved":0,"allocated":61845504,"data_stored":66195434,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":40239104},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673319,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":82876,"kb_used_data":47732,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94288964,"statfs":{"total":96636764160,"available":96551899136,"internally_reserved":0,"allocated":48877568,"data_stored":53464844,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706024,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":82884,"kb_used_data":47740,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94288956,"statfs":{"total":96636764160,"available":96551890944,"internally_reserved":0,"allocated":48885760,"data_stored":52348129,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706023,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":87788,"kb_used_data":52644,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94284052,"statfs":{"total":96636764160,"available":96546869248,"internally_reserved":0,"allocated":53907456,"data_stored":57205636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":48295936,"data_stored":51526038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53379072,"data_stored":56973913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":48287744,"data_stored":52642753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61255680,"data_stored":65373343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T02:01:15.285 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-04-24T02:01:15.286 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2024-04-24T02:01:15.394 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:01:15.387+0000 7f3c2155d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:01:15.398 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:01:15.397+0000 7f3c2155d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:01:15.602 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-04-24T02:01:15.611 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-24T02:01:15.611 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T02:01:15.718 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:01:15.714+0000 7fc1eda2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:01:15.722 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:01:15.721+0000 7fc1eda2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:01:15.905 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T02:01:15.905 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-24T02:01:15.918 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":523,"stamp":"2024-04-24T02:01:15.015551+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":94847753,"num_objects":49,"num_object_clones":14,"num_object_copies":147,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":2572,"num_read_kb":1806935,"num_write":2387,"num_write_kb":611215,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4653,"ondisk_log_size":4653,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":353248,"kb_used_data":208512,"kb_used_omap":0,"kb_used_meta":144704,"kb_avail":377134112,"statfs":{"total":386547056640,"available":386185330688,"internally_reserved":0,"allocated":213516288,"data_stored":229214043,"data_compressed":89889,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":148176896},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.358679"},"pg_stats":[{"pgid":"3.f","version":"342'255","reported_seq":1319,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:12.191911+0000","last_change":"2024-04-24T02:01:00.252850+0000","last_active":"2024-04-24T02:01:12.191911+0000","last_peered":"2024-04-24T02:01:12.191911+0000","last_clean":"2024-04-24T02:01:12.191911+0000","last_became_active":"2024-04-24T01:55:30.386726+0000","last_became_peered":"2024-04-24T01:55:30.386726+0000","last_unstale":"2024-04-24T02:01:12.191911+0000","last_undegraded":"2024-04-24T02:01:12.191911+0000","last_fullsized":"2024-04-24T02:01:12.191911+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":255,"log_dups_size":0,"ondisk_log_size":255,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":23,"snaptrim_duration":0,"stat_sum":{"num_bytes":5512960,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":185,"num_read_kb":165160,"num_write":116,"num_write_kb":33460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"326'374","reported_seq":1370,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:10.682824+0000","last_change":"2024-04-24T02:01:00.196927+0000","last_active":"2024-04-24T02:01:10.682824+0000","last_peered":"2024-04-24T02:01:10.682824+0000","last_clean":"2024-04-24T02:01:10.682824+0000","last_became_active":"2024-04-24T01:55:30.321866+0000","last_became_peered":"2024-04-24T01:55:30.321866+0000","last_unstale":"2024-04-24T02:01:10.682824+0000","last_undegraded":"2024-04-24T02:01:10.682824+0000","last_fullsized":"2024-04-24T02:01:10.682824+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":374,"log_dups_size":0,"ondisk_log_size":374,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":36,"snaptrim_duration":0,"stat_sum":{"num_bytes":2823708,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":203,"num_read_kb":136089,"num_write":185,"num_write_kb":49191,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"342'235","reported_seq":1251,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:12.147307+0000","last_change":"2024-04-24T02:01:00.191968+0000","last_active":"2024-04-24T02:01:12.147307+0000","last_peered":"2024-04-24T02:01:12.147307+0000","last_clean":"2024-04-24T02:01:12.147307+0000","last_became_active":"2024-04-24T01:55:30.320186+0000","last_became_peered":"2024-04-24T01:55:30.320186+0000","last_unstale":"2024-04-24T02:01:12.147307+0000","last_undegraded":"2024-04-24T02:01:12.147307+0000","last_fullsized":"2024-04-24T02:01:12.147307+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":235,"log_dups_size":0,"ondisk_log_size":235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":24,"snaptrim_duration":0,"stat_sum":{"num_bytes":6017104,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":139,"num_read_kb":94253,"num_write":128,"num_write_kb":35473,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"342'309","reported_seq":1347,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:12.192818+0000","last_change":"2024-04-24T02:01:00.248214+0000","last_active":"2024-04-24T02:01:12.192818+0000","last_peered":"2024-04-24T02:01:12.192818+0000","last_clean":"2024-04-24T02:01:12.192818+0000","last_became_active":"2024-04-24T01:55:30.288081+0000","last_became_peered":"2024-04-24T01:55:30.288081+0000","last_unstale":"2024-04-24T02:01:12.192818+0000","last_undegraded":"2024-04-24T02:01:12.192818+0000","last_fullsized":"2024-04-24T02:01:12.192818+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":309,"log_dups_size":0,"ondisk_log_size":309,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":30,"snaptrim_duration":0,"stat_sum":{"num_bytes":9832458,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":161,"num_read_kb":116003,"num_write":154,"num_write_kb":41482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"336'138","reported_seq":1185,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:13.184844+0000","last_change":"2024-04-24T02:00:59.336871+0000","last_active":"2024-04-24T02:01:13.184844+0000","last_peered":"2024-04-24T02:01:13.184844+0000","last_clean":"2024-04-24T02:01:13.184844+0000","last_became_active":"2024-04-24T01:55:30.333815+0000","last_became_peered":"2024-04-24T01:55:30.333815+0000","last_unstale":"2024-04-24T02:01:13.184844+0000","last_undegraded":"2024-04-24T02:01:13.184844+0000","last_fullsized":"2024-04-24T02:01:13.184844+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":17,"snaptrim_duration":0,"stat_sum":{"num_bytes":2857322,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":82,"num_read_kb":52358,"num_write":76,"num_write_kb":20051,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"342'638","reported_seq":1608,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:12.193652+0000","last_change":"2024-04-24T02:01:00.250311+0000","last_active":"2024-04-24T02:01:12.193652+0000","last_peered":"2024-04-24T02:01:12.193652+0000","last_clean":"2024-04-24T02:01:12.193652+0000","last_became_active":"2024-04-24T01:55:30.315075+0000","last_became_peered":"2024-04-24T01:55:30.315075+0000","last_unstale":"2024-04-24T02:01:12.193652+0000","last_undegraded":"2024-04-24T02:01:12.193652+0000","last_fullsized":"2024-04-24T02:01:12.193652+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":638,"log_dups_size":0,"ondisk_log_size":638,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":65,"snaptrim_duration":0,"stat_sum":{"num_bytes":8807264,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":282,"num_read_kb":182263,"num_write":282,"num_write_kb":72883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"342'150","reported_seq":1198,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:12.200097+0000","last_change":"2024-04-24T02:01:00.239736+0000","last_active":"2024-04-24T02:01:12.200097+0000","last_peered":"2024-04-24T02:01:12.200097+0000","last_clean":"2024-04-24T02:01:12.200097+0000","last_became_active":"2024-04-24T01:55:30.350300+0000","last_became_peered":"2024-04-24T01:55:30.350300+0000","last_unstale":"2024-04-24T02:01:12.200097+0000","last_undegraded":"2024-04-24T02:01:12.200097+0000","last_fullsized":"2024-04-24T02:01:12.200097+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":150,"log_dups_size":0,"ondisk_log_size":150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":13,"snaptrim_duration":0,"stat_sum":{"num_bytes":3243883,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":66,"num_read_kb":40460,"num_write":87,"num_write_kb":22401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"264'34","reported_seq":1059,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:12.155738+0000","last_change":"2024-04-24T02:01:00.170396+0000","last_active":"2024-04-24T02:01:12.155738+0000","last_peered":"2024-04-24T02:01:12.155738+0000","last_clean":"2024-04-24T02:01:12.155738+0000","last_became_active":"2024-04-24T01:55:30.376813+0000","last_became_peered":"2024-04-24T01:55:30.376813+0000","last_unstale":"2024-04-24T02:01:12.155738+0000","last_undegraded":"2024-04-24T02:01:12.155738+0000","last_fullsized":"2024-04-24T02:01:12.155738+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0,"stat_sum":{"num_bytes":3025359,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":21,"num_read_kb":15411,"num_write":27,"num_write_kb":6802,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'97","reported_seq":1031,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:13.169480+0000","last_change":"2024-04-24T01:55:16.196470+0000","last_active":"2024-04-24T02:01:13.169480+0000","last_peered":"2024-04-24T02:01:13.169480+0000","last_clean":"2024-04-24T02:01:13.169480+0000","last_became_active":"2024-04-24T01:55:16.190464+0000","last_became_peered":"2024-04-24T01:55:16.190464+0000","last_unstale":"2024-04-24T02:01:13.169480+0000","last_undegraded":"2024-04-24T02:01:13.169480+0000","last_fullsized":"2024-04-24T02:01:13.169480+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:15.127115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:15.127115+0000","last_clean_scrub_stamp":"2024-04-24T01:55:15.127115+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":872,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:12.153789+0000","last_change":"2024-04-24T01:55:18.236148+0000","last_active":"2024-04-24T02:01:12.153789+0000","last_peered":"2024-04-24T02:01:12.153789+0000","last_clean":"2024-04-24T02:01:12.153789+0000","last_became_active":"2024-04-24T01:55:18.230154+0000","last_became_peered":"2024-04-24T01:55:18.230154+0000","last_unstale":"2024-04-24T02:01:12.153789+0000","last_undegraded":"2024-04-24T02:01:12.153789+0000","last_fullsized":"2024-04-24T02:01:12.153789+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"342'462","reported_seq":1458,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:12.153249+0000","last_change":"2024-04-24T02:01:00.337592+0000","last_active":"2024-04-24T02:01:12.153249+0000","last_peered":"2024-04-24T02:01:12.153249+0000","last_clean":"2024-04-24T02:01:12.153249+0000","last_became_active":"2024-04-24T01:55:30.316222+0000","last_became_peered":"2024-04-24T01:55:30.316222+0000","last_unstale":"2024-04-24T02:01:12.153249+0000","last_undegraded":"2024-04-24T02:01:12.153249+0000","last_fullsized":"2024-04-24T02:01:12.153249+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":462,"log_dups_size":0,"ondisk_log_size":462,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":39,"snaptrim_duration":0,"stat_sum":{"num_bytes":3025392,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":231,"num_read_kb":163233,"num_write":236,"num_write_kb":65892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":908,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:12.201364+0000","last_change":"2024-04-24T01:55:18.265905+0000","last_active":"2024-04-24T02:01:12.201364+0000","last_peered":"2024-04-24T02:01:12.201364+0000","last_clean":"2024-04-24T02:01:12.201364+0000","last_became_active":"2024-04-24T01:55:18.257020+0000","last_became_peered":"2024-04-24T01:55:18.257020+0000","last_unstale":"2024-04-24T02:01:12.201364+0000","last_undegraded":"2024-04-24T02:01:12.201364+0000","last_fullsized":"2024-04-24T02:01:12.201364+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"342'358","reported_seq":1367,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:13.173644+0000","last_change":"2024-04-24T02:00:59.359790+0000","last_active":"2024-04-24T02:01:13.173644+0000","last_peered":"2024-04-24T02:01:13.173644+0000","last_clean":"2024-04-24T02:01:13.173644+0000","last_became_active":"2024-04-24T01:55:30.344129+0000","last_became_peered":"2024-04-24T01:55:30.344129+0000","last_unstale":"2024-04-24T02:01:13.173644+0000","last_undegraded":"2024-04-24T02:01:13.173644+0000","last_fullsized":"2024-04-24T02:01:13.173644+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":358,"log_dups_size":0,"ondisk_log_size":358,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":31,"snaptrim_duration":0,"stat_sum":{"num_bytes":10420703,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":155,"num_read_kb":92133,"num_write":178,"num_write_kb":50302,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":872,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:12.146036+0000","last_change":"2024-04-24T01:55:18.215251+0000","last_active":"2024-04-24T02:01:12.146036+0000","last_peered":"2024-04-24T02:01:12.146036+0000","last_clean":"2024-04-24T02:01:12.146036+0000","last_became_active":"2024-04-24T01:55:18.206605+0000","last_became_peered":"2024-04-24T01:55:18.206605+0000","last_unstale":"2024-04-24T02:01:12.146036+0000","last_undegraded":"2024-04-24T02:01:12.146036+0000","last_fullsized":"2024-04-24T02:01:12.146036+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"342'208","reported_seq":1259,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:13.189650+0000","last_change":"2024-04-24T02:00:59.354665+0000","last_active":"2024-04-24T02:01:13.189650+0000","last_peered":"2024-04-24T02:01:13.189650+0000","last_clean":"2024-04-24T02:01:13.189650+0000","last_became_active":"2024-04-24T01:55:30.401366+0000","last_became_peered":"2024-04-24T01:55:30.401366+0000","last_unstale":"2024-04-24T02:01:13.189650+0000","last_undegraded":"2024-04-24T02:01:13.189650+0000","last_fullsized":"2024-04-24T02:01:13.189650+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":208,"log_dups_size":0,"ondisk_log_size":208,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":18,"snaptrim_duration":0,"stat_sum":{"num_bytes":3327918,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":153,"num_read_kb":115484,"num_write":85,"num_write_kb":21155,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":908,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:12.202080+0000","last_change":"2024-04-24T01:55:18.230893+0000","last_active":"2024-04-24T02:01:12.202080+0000","last_peered":"2024-04-24T02:01:12.202080+0000","last_clean":"2024-04-24T02:01:12.202080+0000","last_became_active":"2024-04-24T01:55:18.225188+0000","last_became_peered":"2024-04-24T01:55:18.225188+0000","last_unstale":"2024-04-24T02:01:12.202080+0000","last_undegraded":"2024-04-24T02:01:12.202080+0000","last_fullsized":"2024-04-24T02:01:12.202080+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"342'294","reported_seq":1287,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:10.678999+0000","last_change":"2024-04-24T02:01:00.246736+0000","last_active":"2024-04-24T02:01:10.678999+0000","last_peered":"2024-04-24T02:01:10.678999+0000","last_clean":"2024-04-24T02:01:10.678999+0000","last_became_active":"2024-04-24T01:55:30.343318+0000","last_became_peered":"2024-04-24T01:55:30.343318+0000","last_unstale":"2024-04-24T02:01:10.678999+0000","last_undegraded":"2024-04-24T02:01:10.678999+0000","last_fullsized":"2024-04-24T02:01:10.678999+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":294,"log_dups_size":0,"ondisk_log_size":294,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":26,"snaptrim_duration":0,"stat_sum":{"num_bytes":10118177,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":153,"num_read_kb":122486,"num_write":138,"num_write_kb":38234,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"342'437","reported_seq":1434,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:12.157896+0000","last_change":"2024-04-24T02:01:00.338179+0000","last_active":"2024-04-24T02:01:12.157896+0000","last_peered":"2024-04-24T02:01:12.157896+0000","last_clean":"2024-04-24T02:01:12.157896+0000","last_became_active":"2024-04-24T01:55:30.345308+0000","last_became_peered":"2024-04-24T01:55:30.345308+0000","last_unstale":"2024-04-24T02:01:12.157896+0000","last_undegraded":"2024-04-24T02:01:12.157896+0000","last_fullsized":"2024-04-24T02:01:12.157896+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":437,"log_dups_size":0,"ondisk_log_size":437,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":37,"snaptrim_duration":0,"stat_sum":{"num_bytes":10050949,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":237,"num_read_kb":172356,"num_write":211,"num_write_kb":56847,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":866,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:10.688889+0000","last_change":"2024-04-24T01:55:18.224238+0000","last_active":"2024-04-24T02:01:10.688889+0000","last_peered":"2024-04-24T02:01:10.688889+0000","last_clean":"2024-04-24T02:01:10.688889+0000","last_became_active":"2024-04-24T01:55:18.219131+0000","last_became_peered":"2024-04-24T01:55:18.219131+0000","last_unstale":"2024-04-24T02:01:10.688889+0000","last_undegraded":"2024-04-24T02:01:10.688889+0000","last_fullsized":"2024-04-24T02:01:10.688889+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"342'103","reported_seq":1116,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:10.693419+0000","last_change":"2024-04-24T02:01:00.249384+0000","last_active":"2024-04-24T02:01:10.693419+0000","last_peered":"2024-04-24T02:01:10.693419+0000","last_clean":"2024-04-24T02:01:10.693419+0000","last_became_active":"2024-04-24T01:55:30.319958+0000","last_became_peered":"2024-04-24T01:55:30.319958+0000","last_unstale":"2024-04-24T02:01:10.693419+0000","last_undegraded":"2024-04-24T02:01:10.693419+0000","last_fullsized":"2024-04-24T02:01:10.693419+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":8,"snaptrim_duration":0,"stat_sum":{"num_bytes":3445567,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":50,"num_read_kb":44680,"num_write":61,"num_write_kb":17461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":898,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:13.179399+0000","last_change":"2024-04-24T01:55:18.235763+0000","last_active":"2024-04-24T02:01:13.179399+0000","last_peered":"2024-04-24T02:01:13.179399+0000","last_clean":"2024-04-24T02:01:13.179399+0000","last_became_active":"2024-04-24T01:55:18.229175+0000","last_became_peered":"2024-04-24T01:55:18.229175+0000","last_unstale":"2024-04-24T02:01:13.179399+0000","last_undegraded":"2024-04-24T02:01:13.179399+0000","last_fullsized":"2024-04-24T02:01:13.179399+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"336'192","reported_seq":1228,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:13.172461+0000","last_change":"2024-04-24T02:00:59.343900+0000","last_active":"2024-04-24T02:01:13.172461+0000","last_peered":"2024-04-24T02:01:13.172461+0000","last_clean":"2024-04-24T02:01:13.172461+0000","last_became_active":"2024-04-24T01:55:30.372690+0000","last_became_peered":"2024-04-24T01:55:30.372690+0000","last_unstale":"2024-04-24T02:01:13.172461+0000","last_undegraded":"2024-04-24T02:01:13.172461+0000","last_fullsized":"2024-04-24T02:01:13.172461+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":192,"log_dups_size":0,"ondisk_log_size":192,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":18,"snaptrim_duration":0,"stat_sum":{"num_bytes":5563381,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":102,"num_read_kb":75985,"num_write":95,"num_write_kb":27048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":866,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:10.694645+0000","last_change":"2024-04-24T01:55:18.278351+0000","last_active":"2024-04-24T02:01:10.694645+0000","last_peered":"2024-04-24T02:01:10.694645+0000","last_clean":"2024-04-24T02:01:10.694645+0000","last_became_active":"2024-04-24T01:55:18.273668+0000","last_became_peered":"2024-04-24T01:55:18.273668+0000","last_unstale":"2024-04-24T02:01:10.694645+0000","last_undegraded":"2024-04-24T02:01:10.694645+0000","last_fullsized":"2024-04-24T02:01:10.694645+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"342'369","reported_seq":1391,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:10.689816+0000","last_change":"2024-04-24T02:01:00.639502+0000","last_active":"2024-04-24T02:01:10.689816+0000","last_peered":"2024-04-24T02:01:10.689816+0000","last_clean":"2024-04-24T02:01:10.689816+0000","last_became_active":"2024-04-24T01:55:30.363123+0000","last_became_peered":"2024-04-24T01:55:30.363123+0000","last_unstale":"2024-04-24T02:01:10.689816+0000","last_undegraded":"2024-04-24T02:01:10.689816+0000","last_fullsized":"2024-04-24T02:01:10.689816+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":369,"log_dups_size":0,"ondisk_log_size":369,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":41,"snaptrim_duration":0,"stat_sum":{"num_bytes":6185240,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":249,"num_read_kb":218490,"num_write":191,"num_write_kb":50639,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":872,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:12.156807+0000","last_change":"2024-04-24T01:55:18.241117+0000","last_active":"2024-04-24T02:01:12.156807+0000","last_peered":"2024-04-24T02:01:12.156807+0000","last_clean":"2024-04-24T02:01:12.156807+0000","last_became_active":"2024-04-24T01:55:18.234313+0000","last_became_peered":"2024-04-24T01:55:18.234313+0000","last_unstale":"2024-04-24T02:01:12.156807+0000","last_undegraded":"2024-04-24T02:01:12.156807+0000","last_fullsized":"2024-04-24T02:01:12.156807+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":94257385,"num_objects":47,"num_object_clones":14,"num_object_copies":141,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":2469,"num_read_kb":1806844,"num_write":2250,"num_write_kb":609321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":211218432,"data_stored":226516047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4556,"ondisk_log_size":4556,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":1}],"osd_stats":[{"osd":3,"up_from":10,"seq":42949673319,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":99700,"kb_used_data":60396,"kb_used_omap":0,"kb_used_meta":39296,"kb_avail":94272140,"statfs":{"total":96636764160,"available":96534671360,"internally_reserved":0,"allocated":61845504,"data_stored":66195434,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":40239104},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673319,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":82876,"kb_used_data":47732,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94288964,"statfs":{"total":96636764160,"available":96551899136,"internally_reserved":0,"allocated":48877568,"data_stored":53464844,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706024,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":82884,"kb_used_data":47740,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94288956,"statfs":{"total":96636764160,"available":96551890944,"internally_reserved":0,"allocated":48885760,"data_stored":52348129,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706023,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":87788,"kb_used_data":52644,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94284052,"statfs":{"total":96636764160,"available":96546869248,"internally_reserved":0,"allocated":53907456,"data_stored":57205636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":48295936,"data_stored":51526038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53379072,"data_stored":56973913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":48287744,"data_stored":52642753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61255680,"data_stored":65373343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T02:01:15.919 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T02:01:16.026 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:01:16.022+0000 7fc346c2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:01:16.030 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:01:16.029+0000 7fc346c2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:01:16.219 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T02:01:16.219 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-24T02:01:16.232 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":523,"stamp":"2024-04-24T02:01:15.015551+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":94847753,"num_objects":49,"num_object_clones":14,"num_object_copies":147,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":2572,"num_read_kb":1806935,"num_write":2387,"num_write_kb":611215,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4653,"ondisk_log_size":4653,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":353248,"kb_used_data":208512,"kb_used_omap":0,"kb_used_meta":144704,"kb_avail":377134112,"statfs":{"total":386547056640,"available":386185330688,"internally_reserved":0,"allocated":213516288,"data_stored":229214043,"data_compressed":89889,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":148176896},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.358679"},"pg_stats":[{"pgid":"3.f","version":"342'255","reported_seq":1319,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:12.191911+0000","last_change":"2024-04-24T02:01:00.252850+0000","last_active":"2024-04-24T02:01:12.191911+0000","last_peered":"2024-04-24T02:01:12.191911+0000","last_clean":"2024-04-24T02:01:12.191911+0000","last_became_active":"2024-04-24T01:55:30.386726+0000","last_became_peered":"2024-04-24T01:55:30.386726+0000","last_unstale":"2024-04-24T02:01:12.191911+0000","last_undegraded":"2024-04-24T02:01:12.191911+0000","last_fullsized":"2024-04-24T02:01:12.191911+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":255,"log_dups_size":0,"ondisk_log_size":255,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":23,"snaptrim_duration":0,"stat_sum":{"num_bytes":5512960,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":185,"num_read_kb":165160,"num_write":116,"num_write_kb":33460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"326'374","reported_seq":1370,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:10.682824+0000","last_change":"2024-04-24T02:01:00.196927+0000","last_active":"2024-04-24T02:01:10.682824+0000","last_peered":"2024-04-24T02:01:10.682824+0000","last_clean":"2024-04-24T02:01:10.682824+0000","last_became_active":"2024-04-24T01:55:30.321866+0000","last_became_peered":"2024-04-24T01:55:30.321866+0000","last_unstale":"2024-04-24T02:01:10.682824+0000","last_undegraded":"2024-04-24T02:01:10.682824+0000","last_fullsized":"2024-04-24T02:01:10.682824+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":374,"log_dups_size":0,"ondisk_log_size":374,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":36,"snaptrim_duration":0,"stat_sum":{"num_bytes":2823708,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":203,"num_read_kb":136089,"num_write":185,"num_write_kb":49191,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"342'235","reported_seq":1251,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:12.147307+0000","last_change":"2024-04-24T02:01:00.191968+0000","last_active":"2024-04-24T02:01:12.147307+0000","last_peered":"2024-04-24T02:01:12.147307+0000","last_clean":"2024-04-24T02:01:12.147307+0000","last_became_active":"2024-04-24T01:55:30.320186+0000","last_became_peered":"2024-04-24T01:55:30.320186+0000","last_unstale":"2024-04-24T02:01:12.147307+0000","last_undegraded":"2024-04-24T02:01:12.147307+0000","last_fullsized":"2024-04-24T02:01:12.147307+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":235,"log_dups_size":0,"ondisk_log_size":235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":24,"snaptrim_duration":0,"stat_sum":{"num_bytes":6017104,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":139,"num_read_kb":94253,"num_write":128,"num_write_kb":35473,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"342'309","reported_seq":1347,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:12.192818+0000","last_change":"2024-04-24T02:01:00.248214+0000","last_active":"2024-04-24T02:01:12.192818+0000","last_peered":"2024-04-24T02:01:12.192818+0000","last_clean":"2024-04-24T02:01:12.192818+0000","last_became_active":"2024-04-24T01:55:30.288081+0000","last_became_peered":"2024-04-24T01:55:30.288081+0000","last_unstale":"2024-04-24T02:01:12.192818+0000","last_undegraded":"2024-04-24T02:01:12.192818+0000","last_fullsized":"2024-04-24T02:01:12.192818+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":309,"log_dups_size":0,"ondisk_log_size":309,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":30,"snaptrim_duration":0,"stat_sum":{"num_bytes":9832458,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":161,"num_read_kb":116003,"num_write":154,"num_write_kb":41482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"336'138","reported_seq":1185,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:13.184844+0000","last_change":"2024-04-24T02:00:59.336871+0000","last_active":"2024-04-24T02:01:13.184844+0000","last_peered":"2024-04-24T02:01:13.184844+0000","last_clean":"2024-04-24T02:01:13.184844+0000","last_became_active":"2024-04-24T01:55:30.333815+0000","last_became_peered":"2024-04-24T01:55:30.333815+0000","last_unstale":"2024-04-24T02:01:13.184844+0000","last_undegraded":"2024-04-24T02:01:13.184844+0000","last_fullsized":"2024-04-24T02:01:13.184844+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":17,"snaptrim_duration":0,"stat_sum":{"num_bytes":2857322,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":82,"num_read_kb":52358,"num_write":76,"num_write_kb":20051,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"342'638","reported_seq":1608,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:12.193652+0000","last_change":"2024-04-24T02:01:00.250311+0000","last_active":"2024-04-24T02:01:12.193652+0000","last_peered":"2024-04-24T02:01:12.193652+0000","last_clean":"2024-04-24T02:01:12.193652+0000","last_became_active":"2024-04-24T01:55:30.315075+0000","last_became_peered":"2024-04-24T01:55:30.315075+0000","last_unstale":"2024-04-24T02:01:12.193652+0000","last_undegraded":"2024-04-24T02:01:12.193652+0000","last_fullsized":"2024-04-24T02:01:12.193652+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":638,"log_dups_size":0,"ondisk_log_size":638,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":65,"snaptrim_duration":0,"stat_sum":{"num_bytes":8807264,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":282,"num_read_kb":182263,"num_write":282,"num_write_kb":72883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"342'150","reported_seq":1198,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:12.200097+0000","last_change":"2024-04-24T02:01:00.239736+0000","last_active":"2024-04-24T02:01:12.200097+0000","last_peered":"2024-04-24T02:01:12.200097+0000","last_clean":"2024-04-24T02:01:12.200097+0000","last_became_active":"2024-04-24T01:55:30.350300+0000","last_became_peered":"2024-04-24T01:55:30.350300+0000","last_unstale":"2024-04-24T02:01:12.200097+0000","last_undegraded":"2024-04-24T02:01:12.200097+0000","last_fullsized":"2024-04-24T02:01:12.200097+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":150,"log_dups_size":0,"ondisk_log_size":150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":13,"snaptrim_duration":0,"stat_sum":{"num_bytes":3243883,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":66,"num_read_kb":40460,"num_write":87,"num_write_kb":22401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"264'34","reported_seq":1059,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:12.155738+0000","last_change":"2024-04-24T02:01:00.170396+0000","last_active":"2024-04-24T02:01:12.155738+0000","last_peered":"2024-04-24T02:01:12.155738+0000","last_clean":"2024-04-24T02:01:12.155738+0000","last_became_active":"2024-04-24T01:55:30.376813+0000","last_became_peered":"2024-04-24T01:55:30.376813+0000","last_unstale":"2024-04-24T02:01:12.155738+0000","last_undegraded":"2024-04-24T02:01:12.155738+0000","last_fullsized":"2024-04-24T02:01:12.155738+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0,"stat_sum":{"num_bytes":3025359,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":21,"num_read_kb":15411,"num_write":27,"num_write_kb":6802,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'97","reported_seq":1031,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:13.169480+0000","last_change":"2024-04-24T01:55:16.196470+0000","last_active":"2024-04-24T02:01:13.169480+0000","last_peered":"2024-04-24T02:01:13.169480+0000","last_clean":"2024-04-24T02:01:13.169480+0000","last_became_active":"2024-04-24T01:55:16.190464+0000","last_became_peered":"2024-04-24T01:55:16.190464+0000","last_unstale":"2024-04-24T02:01:13.169480+0000","last_undegraded":"2024-04-24T02:01:13.169480+0000","last_fullsized":"2024-04-24T02:01:13.169480+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:15.127115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:15.127115+0000","last_clean_scrub_stamp":"2024-04-24T01:55:15.127115+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":872,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:12.153789+0000","last_change":"2024-04-24T01:55:18.236148+0000","last_active":"2024-04-24T02:01:12.153789+0000","last_peered":"2024-04-24T02:01:12.153789+0000","last_clean":"2024-04-24T02:01:12.153789+0000","last_became_active":"2024-04-24T01:55:18.230154+0000","last_became_peered":"2024-04-24T01:55:18.230154+0000","last_unstale":"2024-04-24T02:01:12.153789+0000","last_undegraded":"2024-04-24T02:01:12.153789+0000","last_fullsized":"2024-04-24T02:01:12.153789+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"342'462","reported_seq":1458,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:12.153249+0000","last_change":"2024-04-24T02:01:00.337592+0000","last_active":"2024-04-24T02:01:12.153249+0000","last_peered":"2024-04-24T02:01:12.153249+0000","last_clean":"2024-04-24T02:01:12.153249+0000","last_became_active":"2024-04-24T01:55:30.316222+0000","last_became_peered":"2024-04-24T01:55:30.316222+0000","last_unstale":"2024-04-24T02:01:12.153249+0000","last_undegraded":"2024-04-24T02:01:12.153249+0000","last_fullsized":"2024-04-24T02:01:12.153249+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":462,"log_dups_size":0,"ondisk_log_size":462,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":39,"snaptrim_duration":0,"stat_sum":{"num_bytes":3025392,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":231,"num_read_kb":163233,"num_write":236,"num_write_kb":65892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":908,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:12.201364+0000","last_change":"2024-04-24T01:55:18.265905+0000","last_active":"2024-04-24T02:01:12.201364+0000","last_peered":"2024-04-24T02:01:12.201364+0000","last_clean":"2024-04-24T02:01:12.201364+0000","last_became_active":"2024-04-24T01:55:18.257020+0000","last_became_peered":"2024-04-24T01:55:18.257020+0000","last_unstale":"2024-04-24T02:01:12.201364+0000","last_undegraded":"2024-04-24T02:01:12.201364+0000","last_fullsized":"2024-04-24T02:01:12.201364+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"342'358","reported_seq":1367,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:13.173644+0000","last_change":"2024-04-24T02:00:59.359790+0000","last_active":"2024-04-24T02:01:13.173644+0000","last_peered":"2024-04-24T02:01:13.173644+0000","last_clean":"2024-04-24T02:01:13.173644+0000","last_became_active":"2024-04-24T01:55:30.344129+0000","last_became_peered":"2024-04-24T01:55:30.344129+0000","last_unstale":"2024-04-24T02:01:13.173644+0000","last_undegraded":"2024-04-24T02:01:13.173644+0000","last_fullsized":"2024-04-24T02:01:13.173644+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":358,"log_dups_size":0,"ondisk_log_size":358,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":31,"snaptrim_duration":0,"stat_sum":{"num_bytes":10420703,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":155,"num_read_kb":92133,"num_write":178,"num_write_kb":50302,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":872,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:12.146036+0000","last_change":"2024-04-24T01:55:18.215251+0000","last_active":"2024-04-24T02:01:12.146036+0000","last_peered":"2024-04-24T02:01:12.146036+0000","last_clean":"2024-04-24T02:01:12.146036+0000","last_became_active":"2024-04-24T01:55:18.206605+0000","last_became_peered":"2024-04-24T01:55:18.206605+0000","last_unstale":"2024-04-24T02:01:12.146036+0000","last_undegraded":"2024-04-24T02:01:12.146036+0000","last_fullsized":"2024-04-24T02:01:12.146036+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"342'208","reported_seq":1259,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:13.189650+0000","last_change":"2024-04-24T02:00:59.354665+0000","last_active":"2024-04-24T02:01:13.189650+0000","last_peered":"2024-04-24T02:01:13.189650+0000","last_clean":"2024-04-24T02:01:13.189650+0000","last_became_active":"2024-04-24T01:55:30.401366+0000","last_became_peered":"2024-04-24T01:55:30.401366+0000","last_unstale":"2024-04-24T02:01:13.189650+0000","last_undegraded":"2024-04-24T02:01:13.189650+0000","last_fullsized":"2024-04-24T02:01:13.189650+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":208,"log_dups_size":0,"ondisk_log_size":208,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":18,"snaptrim_duration":0,"stat_sum":{"num_bytes":3327918,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":153,"num_read_kb":115484,"num_write":85,"num_write_kb":21155,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":908,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:12.202080+0000","last_change":"2024-04-24T01:55:18.230893+0000","last_active":"2024-04-24T02:01:12.202080+0000","last_peered":"2024-04-24T02:01:12.202080+0000","last_clean":"2024-04-24T02:01:12.202080+0000","last_became_active":"2024-04-24T01:55:18.225188+0000","last_became_peered":"2024-04-24T01:55:18.225188+0000","last_unstale":"2024-04-24T02:01:12.202080+0000","last_undegraded":"2024-04-24T02:01:12.202080+0000","last_fullsized":"2024-04-24T02:01:12.202080+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"342'294","reported_seq":1287,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:10.678999+0000","last_change":"2024-04-24T02:01:00.246736+0000","last_active":"2024-04-24T02:01:10.678999+0000","last_peered":"2024-04-24T02:01:10.678999+0000","last_clean":"2024-04-24T02:01:10.678999+0000","last_became_active":"2024-04-24T01:55:30.343318+0000","last_became_peered":"2024-04-24T01:55:30.343318+0000","last_unstale":"2024-04-24T02:01:10.678999+0000","last_undegraded":"2024-04-24T02:01:10.678999+0000","last_fullsized":"2024-04-24T02:01:10.678999+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":294,"log_dups_size":0,"ondisk_log_size":294,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":26,"snaptrim_duration":0,"stat_sum":{"num_bytes":10118177,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":153,"num_read_kb":122486,"num_write":138,"num_write_kb":38234,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"342'437","reported_seq":1434,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:12.157896+0000","last_change":"2024-04-24T02:01:00.338179+0000","last_active":"2024-04-24T02:01:12.157896+0000","last_peered":"2024-04-24T02:01:12.157896+0000","last_clean":"2024-04-24T02:01:12.157896+0000","last_became_active":"2024-04-24T01:55:30.345308+0000","last_became_peered":"2024-04-24T01:55:30.345308+0000","last_unstale":"2024-04-24T02:01:12.157896+0000","last_undegraded":"2024-04-24T02:01:12.157896+0000","last_fullsized":"2024-04-24T02:01:12.157896+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":437,"log_dups_size":0,"ondisk_log_size":437,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":37,"snaptrim_duration":0,"stat_sum":{"num_bytes":10050949,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":237,"num_read_kb":172356,"num_write":211,"num_write_kb":56847,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":866,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:10.688889+0000","last_change":"2024-04-24T01:55:18.224238+0000","last_active":"2024-04-24T02:01:10.688889+0000","last_peered":"2024-04-24T02:01:10.688889+0000","last_clean":"2024-04-24T02:01:10.688889+0000","last_became_active":"2024-04-24T01:55:18.219131+0000","last_became_peered":"2024-04-24T01:55:18.219131+0000","last_unstale":"2024-04-24T02:01:10.688889+0000","last_undegraded":"2024-04-24T02:01:10.688889+0000","last_fullsized":"2024-04-24T02:01:10.688889+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"342'103","reported_seq":1116,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:10.693419+0000","last_change":"2024-04-24T02:01:00.249384+0000","last_active":"2024-04-24T02:01:10.693419+0000","last_peered":"2024-04-24T02:01:10.693419+0000","last_clean":"2024-04-24T02:01:10.693419+0000","last_became_active":"2024-04-24T01:55:30.319958+0000","last_became_peered":"2024-04-24T01:55:30.319958+0000","last_unstale":"2024-04-24T02:01:10.693419+0000","last_undegraded":"2024-04-24T02:01:10.693419+0000","last_fullsized":"2024-04-24T02:01:10.693419+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":8,"snaptrim_duration":0,"stat_sum":{"num_bytes":3445567,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":50,"num_read_kb":44680,"num_write":61,"num_write_kb":17461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":898,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:13.179399+0000","last_change":"2024-04-24T01:55:18.235763+0000","last_active":"2024-04-24T02:01:13.179399+0000","last_peered":"2024-04-24T02:01:13.179399+0000","last_clean":"2024-04-24T02:01:13.179399+0000","last_became_active":"2024-04-24T01:55:18.229175+0000","last_became_peered":"2024-04-24T01:55:18.229175+0000","last_unstale":"2024-04-24T02:01:13.179399+0000","last_undegraded":"2024-04-24T02:01:13.179399+0000","last_fullsized":"2024-04-24T02:01:13.179399+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"336'192","reported_seq":1228,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:13.172461+0000","last_change":"2024-04-24T02:00:59.343900+0000","last_active":"2024-04-24T02:01:13.172461+0000","last_peered":"2024-04-24T02:01:13.172461+0000","last_clean":"2024-04-24T02:01:13.172461+0000","last_became_active":"2024-04-24T01:55:30.372690+0000","last_became_peered":"2024-04-24T01:55:30.372690+0000","last_unstale":"2024-04-24T02:01:13.172461+0000","last_undegraded":"2024-04-24T02:01:13.172461+0000","last_fullsized":"2024-04-24T02:01:13.172461+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":192,"log_dups_size":0,"ondisk_log_size":192,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":18,"snaptrim_duration":0,"stat_sum":{"num_bytes":5563381,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":102,"num_read_kb":75985,"num_write":95,"num_write_kb":27048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":866,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:10.694645+0000","last_change":"2024-04-24T01:55:18.278351+0000","last_active":"2024-04-24T02:01:10.694645+0000","last_peered":"2024-04-24T02:01:10.694645+0000","last_clean":"2024-04-24T02:01:10.694645+0000","last_became_active":"2024-04-24T01:55:18.273668+0000","last_became_peered":"2024-04-24T01:55:18.273668+0000","last_unstale":"2024-04-24T02:01:10.694645+0000","last_undegraded":"2024-04-24T02:01:10.694645+0000","last_fullsized":"2024-04-24T02:01:10.694645+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"342'369","reported_seq":1391,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:10.689816+0000","last_change":"2024-04-24T02:01:00.639502+0000","last_active":"2024-04-24T02:01:10.689816+0000","last_peered":"2024-04-24T02:01:10.689816+0000","last_clean":"2024-04-24T02:01:10.689816+0000","last_became_active":"2024-04-24T01:55:30.363123+0000","last_became_peered":"2024-04-24T01:55:30.363123+0000","last_unstale":"2024-04-24T02:01:10.689816+0000","last_undegraded":"2024-04-24T02:01:10.689816+0000","last_fullsized":"2024-04-24T02:01:10.689816+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":369,"log_dups_size":0,"ondisk_log_size":369,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":41,"snaptrim_duration":0,"stat_sum":{"num_bytes":6185240,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":249,"num_read_kb":218490,"num_write":191,"num_write_kb":50639,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":872,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:12.156807+0000","last_change":"2024-04-24T01:55:18.241117+0000","last_active":"2024-04-24T02:01:12.156807+0000","last_peered":"2024-04-24T02:01:12.156807+0000","last_clean":"2024-04-24T02:01:12.156807+0000","last_became_active":"2024-04-24T01:55:18.234313+0000","last_became_peered":"2024-04-24T01:55:18.234313+0000","last_unstale":"2024-04-24T02:01:12.156807+0000","last_undegraded":"2024-04-24T02:01:12.156807+0000","last_fullsized":"2024-04-24T02:01:12.156807+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":94257385,"num_objects":47,"num_object_clones":14,"num_object_copies":141,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":2469,"num_read_kb":1806844,"num_write":2250,"num_write_kb":609321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":211218432,"data_stored":226516047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4556,"ondisk_log_size":4556,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":1}],"osd_stats":[{"osd":3,"up_from":10,"seq":42949673319,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":99700,"kb_used_data":60396,"kb_used_omap":0,"kb_used_meta":39296,"kb_avail":94272140,"statfs":{"total":96636764160,"available":96534671360,"internally_reserved":0,"allocated":61845504,"data_stored":66195434,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":40239104},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673319,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":82876,"kb_used_data":47732,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94288964,"statfs":{"total":96636764160,"available":96551899136,"internally_reserved":0,"allocated":48877568,"data_stored":53464844,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706024,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":82884,"kb_used_data":47740,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94288956,"statfs":{"total":96636764160,"available":96551890944,"internally_reserved":0,"allocated":48885760,"data_stored":52348129,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706023,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":87788,"kb_used_data":52644,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94284052,"statfs":{"total":96636764160,"available":96546869248,"internally_reserved":0,"allocated":53907456,"data_stored":57205636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":48295936,"data_stored":51526038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53379072,"data_stored":56973913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":48287744,"data_stored":52642753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61255680,"data_stored":65373343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T02:01:16.233 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-24T02:01:16.233 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T02:01:16.341 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:01:16.337+0000 7f9c9802d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:01:16.345 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:01:16.344+0000 7f9c9802d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:01:16.531 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T02:01:16.531 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-24T02:01:16.544 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":523,"stamp":"2024-04-24T02:01:15.015551+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":94847753,"num_objects":49,"num_object_clones":14,"num_object_copies":147,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":2572,"num_read_kb":1806935,"num_write":2387,"num_write_kb":611215,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4653,"ondisk_log_size":4653,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":353248,"kb_used_data":208512,"kb_used_omap":0,"kb_used_meta":144704,"kb_avail":377134112,"statfs":{"total":386547056640,"available":386185330688,"internally_reserved":0,"allocated":213516288,"data_stored":229214043,"data_compressed":89889,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":148176896},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.358679"},"pg_stats":[{"pgid":"3.f","version":"342'255","reported_seq":1319,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:12.191911+0000","last_change":"2024-04-24T02:01:00.252850+0000","last_active":"2024-04-24T02:01:12.191911+0000","last_peered":"2024-04-24T02:01:12.191911+0000","last_clean":"2024-04-24T02:01:12.191911+0000","last_became_active":"2024-04-24T01:55:30.386726+0000","last_became_peered":"2024-04-24T01:55:30.386726+0000","last_unstale":"2024-04-24T02:01:12.191911+0000","last_undegraded":"2024-04-24T02:01:12.191911+0000","last_fullsized":"2024-04-24T02:01:12.191911+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":255,"log_dups_size":0,"ondisk_log_size":255,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":23,"snaptrim_duration":0,"stat_sum":{"num_bytes":5512960,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":185,"num_read_kb":165160,"num_write":116,"num_write_kb":33460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"326'374","reported_seq":1370,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:10.682824+0000","last_change":"2024-04-24T02:01:00.196927+0000","last_active":"2024-04-24T02:01:10.682824+0000","last_peered":"2024-04-24T02:01:10.682824+0000","last_clean":"2024-04-24T02:01:10.682824+0000","last_became_active":"2024-04-24T01:55:30.321866+0000","last_became_peered":"2024-04-24T01:55:30.321866+0000","last_unstale":"2024-04-24T02:01:10.682824+0000","last_undegraded":"2024-04-24T02:01:10.682824+0000","last_fullsized":"2024-04-24T02:01:10.682824+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":374,"log_dups_size":0,"ondisk_log_size":374,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":36,"snaptrim_duration":0,"stat_sum":{"num_bytes":2823708,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":203,"num_read_kb":136089,"num_write":185,"num_write_kb":49191,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"342'235","reported_seq":1251,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:12.147307+0000","last_change":"2024-04-24T02:01:00.191968+0000","last_active":"2024-04-24T02:01:12.147307+0000","last_peered":"2024-04-24T02:01:12.147307+0000","last_clean":"2024-04-24T02:01:12.147307+0000","last_became_active":"2024-04-24T01:55:30.320186+0000","last_became_peered":"2024-04-24T01:55:30.320186+0000","last_unstale":"2024-04-24T02:01:12.147307+0000","last_undegraded":"2024-04-24T02:01:12.147307+0000","last_fullsized":"2024-04-24T02:01:12.147307+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":235,"log_dups_size":0,"ondisk_log_size":235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":24,"snaptrim_duration":0,"stat_sum":{"num_bytes":6017104,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":139,"num_read_kb":94253,"num_write":128,"num_write_kb":35473,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"342'309","reported_seq":1347,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:12.192818+0000","last_change":"2024-04-24T02:01:00.248214+0000","last_active":"2024-04-24T02:01:12.192818+0000","last_peered":"2024-04-24T02:01:12.192818+0000","last_clean":"2024-04-24T02:01:12.192818+0000","last_became_active":"2024-04-24T01:55:30.288081+0000","last_became_peered":"2024-04-24T01:55:30.288081+0000","last_unstale":"2024-04-24T02:01:12.192818+0000","last_undegraded":"2024-04-24T02:01:12.192818+0000","last_fullsized":"2024-04-24T02:01:12.192818+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":309,"log_dups_size":0,"ondisk_log_size":309,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":30,"snaptrim_duration":0,"stat_sum":{"num_bytes":9832458,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":161,"num_read_kb":116003,"num_write":154,"num_write_kb":41482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"336'138","reported_seq":1185,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:13.184844+0000","last_change":"2024-04-24T02:00:59.336871+0000","last_active":"2024-04-24T02:01:13.184844+0000","last_peered":"2024-04-24T02:01:13.184844+0000","last_clean":"2024-04-24T02:01:13.184844+0000","last_became_active":"2024-04-24T01:55:30.333815+0000","last_became_peered":"2024-04-24T01:55:30.333815+0000","last_unstale":"2024-04-24T02:01:13.184844+0000","last_undegraded":"2024-04-24T02:01:13.184844+0000","last_fullsized":"2024-04-24T02:01:13.184844+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":17,"snaptrim_duration":0,"stat_sum":{"num_bytes":2857322,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":82,"num_read_kb":52358,"num_write":76,"num_write_kb":20051,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"342'638","reported_seq":1608,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:12.193652+0000","last_change":"2024-04-24T02:01:00.250311+0000","last_active":"2024-04-24T02:01:12.193652+0000","last_peered":"2024-04-24T02:01:12.193652+0000","last_clean":"2024-04-24T02:01:12.193652+0000","last_became_active":"2024-04-24T01:55:30.315075+0000","last_became_peered":"2024-04-24T01:55:30.315075+0000","last_unstale":"2024-04-24T02:01:12.193652+0000","last_undegraded":"2024-04-24T02:01:12.193652+0000","last_fullsized":"2024-04-24T02:01:12.193652+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":638,"log_dups_size":0,"ondisk_log_size":638,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":65,"snaptrim_duration":0,"stat_sum":{"num_bytes":8807264,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":282,"num_read_kb":182263,"num_write":282,"num_write_kb":72883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"342'150","reported_seq":1198,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:12.200097+0000","last_change":"2024-04-24T02:01:00.239736+0000","last_active":"2024-04-24T02:01:12.200097+0000","last_peered":"2024-04-24T02:01:12.200097+0000","last_clean":"2024-04-24T02:01:12.200097+0000","last_became_active":"2024-04-24T01:55:30.350300+0000","last_became_peered":"2024-04-24T01:55:30.350300+0000","last_unstale":"2024-04-24T02:01:12.200097+0000","last_undegraded":"2024-04-24T02:01:12.200097+0000","last_fullsized":"2024-04-24T02:01:12.200097+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":150,"log_dups_size":0,"ondisk_log_size":150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":13,"snaptrim_duration":0,"stat_sum":{"num_bytes":3243883,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":66,"num_read_kb":40460,"num_write":87,"num_write_kb":22401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"264'34","reported_seq":1059,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:12.155738+0000","last_change":"2024-04-24T02:01:00.170396+0000","last_active":"2024-04-24T02:01:12.155738+0000","last_peered":"2024-04-24T02:01:12.155738+0000","last_clean":"2024-04-24T02:01:12.155738+0000","last_became_active":"2024-04-24T01:55:30.376813+0000","last_became_peered":"2024-04-24T01:55:30.376813+0000","last_unstale":"2024-04-24T02:01:12.155738+0000","last_undegraded":"2024-04-24T02:01:12.155738+0000","last_fullsized":"2024-04-24T02:01:12.155738+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0,"stat_sum":{"num_bytes":3025359,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":21,"num_read_kb":15411,"num_write":27,"num_write_kb":6802,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'97","reported_seq":1031,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:13.169480+0000","last_change":"2024-04-24T01:55:16.196470+0000","last_active":"2024-04-24T02:01:13.169480+0000","last_peered":"2024-04-24T02:01:13.169480+0000","last_clean":"2024-04-24T02:01:13.169480+0000","last_became_active":"2024-04-24T01:55:16.190464+0000","last_became_peered":"2024-04-24T01:55:16.190464+0000","last_unstale":"2024-04-24T02:01:13.169480+0000","last_undegraded":"2024-04-24T02:01:13.169480+0000","last_fullsized":"2024-04-24T02:01:13.169480+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:15.127115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:15.127115+0000","last_clean_scrub_stamp":"2024-04-24T01:55:15.127115+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":872,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:12.153789+0000","last_change":"2024-04-24T01:55:18.236148+0000","last_active":"2024-04-24T02:01:12.153789+0000","last_peered":"2024-04-24T02:01:12.153789+0000","last_clean":"2024-04-24T02:01:12.153789+0000","last_became_active":"2024-04-24T01:55:18.230154+0000","last_became_peered":"2024-04-24T01:55:18.230154+0000","last_unstale":"2024-04-24T02:01:12.153789+0000","last_undegraded":"2024-04-24T02:01:12.153789+0000","last_fullsized":"2024-04-24T02:01:12.153789+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"342'462","reported_seq":1458,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:12.153249+0000","last_change":"2024-04-24T02:01:00.337592+0000","last_active":"2024-04-24T02:01:12.153249+0000","last_peered":"2024-04-24T02:01:12.153249+0000","last_clean":"2024-04-24T02:01:12.153249+0000","last_became_active":"2024-04-24T01:55:30.316222+0000","last_became_peered":"2024-04-24T01:55:30.316222+0000","last_unstale":"2024-04-24T02:01:12.153249+0000","last_undegraded":"2024-04-24T02:01:12.153249+0000","last_fullsized":"2024-04-24T02:01:12.153249+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":462,"log_dups_size":0,"ondisk_log_size":462,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":39,"snaptrim_duration":0,"stat_sum":{"num_bytes":3025392,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":231,"num_read_kb":163233,"num_write":236,"num_write_kb":65892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":908,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:12.201364+0000","last_change":"2024-04-24T01:55:18.265905+0000","last_active":"2024-04-24T02:01:12.201364+0000","last_peered":"2024-04-24T02:01:12.201364+0000","last_clean":"2024-04-24T02:01:12.201364+0000","last_became_active":"2024-04-24T01:55:18.257020+0000","last_became_peered":"2024-04-24T01:55:18.257020+0000","last_unstale":"2024-04-24T02:01:12.201364+0000","last_undegraded":"2024-04-24T02:01:12.201364+0000","last_fullsized":"2024-04-24T02:01:12.201364+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"342'358","reported_seq":1367,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:13.173644+0000","last_change":"2024-04-24T02:00:59.359790+0000","last_active":"2024-04-24T02:01:13.173644+0000","last_peered":"2024-04-24T02:01:13.173644+0000","last_clean":"2024-04-24T02:01:13.173644+0000","last_became_active":"2024-04-24T01:55:30.344129+0000","last_became_peered":"2024-04-24T01:55:30.344129+0000","last_unstale":"2024-04-24T02:01:13.173644+0000","last_undegraded":"2024-04-24T02:01:13.173644+0000","last_fullsized":"2024-04-24T02:01:13.173644+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":358,"log_dups_size":0,"ondisk_log_size":358,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":31,"snaptrim_duration":0,"stat_sum":{"num_bytes":10420703,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":155,"num_read_kb":92133,"num_write":178,"num_write_kb":50302,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":872,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:12.146036+0000","last_change":"2024-04-24T01:55:18.215251+0000","last_active":"2024-04-24T02:01:12.146036+0000","last_peered":"2024-04-24T02:01:12.146036+0000","last_clean":"2024-04-24T02:01:12.146036+0000","last_became_active":"2024-04-24T01:55:18.206605+0000","last_became_peered":"2024-04-24T01:55:18.206605+0000","last_unstale":"2024-04-24T02:01:12.146036+0000","last_undegraded":"2024-04-24T02:01:12.146036+0000","last_fullsized":"2024-04-24T02:01:12.146036+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"342'208","reported_seq":1259,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:13.189650+0000","last_change":"2024-04-24T02:00:59.354665+0000","last_active":"2024-04-24T02:01:13.189650+0000","last_peered":"2024-04-24T02:01:13.189650+0000","last_clean":"2024-04-24T02:01:13.189650+0000","last_became_active":"2024-04-24T01:55:30.401366+0000","last_became_peered":"2024-04-24T01:55:30.401366+0000","last_unstale":"2024-04-24T02:01:13.189650+0000","last_undegraded":"2024-04-24T02:01:13.189650+0000","last_fullsized":"2024-04-24T02:01:13.189650+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":208,"log_dups_size":0,"ondisk_log_size":208,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":18,"snaptrim_duration":0,"stat_sum":{"num_bytes":3327918,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":153,"num_read_kb":115484,"num_write":85,"num_write_kb":21155,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":908,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:12.202080+0000","last_change":"2024-04-24T01:55:18.230893+0000","last_active":"2024-04-24T02:01:12.202080+0000","last_peered":"2024-04-24T02:01:12.202080+0000","last_clean":"2024-04-24T02:01:12.202080+0000","last_became_active":"2024-04-24T01:55:18.225188+0000","last_became_peered":"2024-04-24T01:55:18.225188+0000","last_unstale":"2024-04-24T02:01:12.202080+0000","last_undegraded":"2024-04-24T02:01:12.202080+0000","last_fullsized":"2024-04-24T02:01:12.202080+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"342'294","reported_seq":1287,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:10.678999+0000","last_change":"2024-04-24T02:01:00.246736+0000","last_active":"2024-04-24T02:01:10.678999+0000","last_peered":"2024-04-24T02:01:10.678999+0000","last_clean":"2024-04-24T02:01:10.678999+0000","last_became_active":"2024-04-24T01:55:30.343318+0000","last_became_peered":"2024-04-24T01:55:30.343318+0000","last_unstale":"2024-04-24T02:01:10.678999+0000","last_undegraded":"2024-04-24T02:01:10.678999+0000","last_fullsized":"2024-04-24T02:01:10.678999+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":294,"log_dups_size":0,"ondisk_log_size":294,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":26,"snaptrim_duration":0,"stat_sum":{"num_bytes":10118177,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":153,"num_read_kb":122486,"num_write":138,"num_write_kb":38234,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"342'437","reported_seq":1434,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:12.157896+0000","last_change":"2024-04-24T02:01:00.338179+0000","last_active":"2024-04-24T02:01:12.157896+0000","last_peered":"2024-04-24T02:01:12.157896+0000","last_clean":"2024-04-24T02:01:12.157896+0000","last_became_active":"2024-04-24T01:55:30.345308+0000","last_became_peered":"2024-04-24T01:55:30.345308+0000","last_unstale":"2024-04-24T02:01:12.157896+0000","last_undegraded":"2024-04-24T02:01:12.157896+0000","last_fullsized":"2024-04-24T02:01:12.157896+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":437,"log_dups_size":0,"ondisk_log_size":437,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":37,"snaptrim_duration":0,"stat_sum":{"num_bytes":10050949,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":237,"num_read_kb":172356,"num_write":211,"num_write_kb":56847,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":866,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:10.688889+0000","last_change":"2024-04-24T01:55:18.224238+0000","last_active":"2024-04-24T02:01:10.688889+0000","last_peered":"2024-04-24T02:01:10.688889+0000","last_clean":"2024-04-24T02:01:10.688889+0000","last_became_active":"2024-04-24T01:55:18.219131+0000","last_became_peered":"2024-04-24T01:55:18.219131+0000","last_unstale":"2024-04-24T02:01:10.688889+0000","last_undegraded":"2024-04-24T02:01:10.688889+0000","last_fullsized":"2024-04-24T02:01:10.688889+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"342'103","reported_seq":1116,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:10.693419+0000","last_change":"2024-04-24T02:01:00.249384+0000","last_active":"2024-04-24T02:01:10.693419+0000","last_peered":"2024-04-24T02:01:10.693419+0000","last_clean":"2024-04-24T02:01:10.693419+0000","last_became_active":"2024-04-24T01:55:30.319958+0000","last_became_peered":"2024-04-24T01:55:30.319958+0000","last_unstale":"2024-04-24T02:01:10.693419+0000","last_undegraded":"2024-04-24T02:01:10.693419+0000","last_fullsized":"2024-04-24T02:01:10.693419+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":8,"snaptrim_duration":0,"stat_sum":{"num_bytes":3445567,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":50,"num_read_kb":44680,"num_write":61,"num_write_kb":17461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":898,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:13.179399+0000","last_change":"2024-04-24T01:55:18.235763+0000","last_active":"2024-04-24T02:01:13.179399+0000","last_peered":"2024-04-24T02:01:13.179399+0000","last_clean":"2024-04-24T02:01:13.179399+0000","last_became_active":"2024-04-24T01:55:18.229175+0000","last_became_peered":"2024-04-24T01:55:18.229175+0000","last_unstale":"2024-04-24T02:01:13.179399+0000","last_undegraded":"2024-04-24T02:01:13.179399+0000","last_fullsized":"2024-04-24T02:01:13.179399+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"336'192","reported_seq":1228,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:13.172461+0000","last_change":"2024-04-24T02:00:59.343900+0000","last_active":"2024-04-24T02:01:13.172461+0000","last_peered":"2024-04-24T02:01:13.172461+0000","last_clean":"2024-04-24T02:01:13.172461+0000","last_became_active":"2024-04-24T01:55:30.372690+0000","last_became_peered":"2024-04-24T01:55:30.372690+0000","last_unstale":"2024-04-24T02:01:13.172461+0000","last_undegraded":"2024-04-24T02:01:13.172461+0000","last_fullsized":"2024-04-24T02:01:13.172461+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":192,"log_dups_size":0,"ondisk_log_size":192,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":18,"snaptrim_duration":0,"stat_sum":{"num_bytes":5563381,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":102,"num_read_kb":75985,"num_write":95,"num_write_kb":27048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":866,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:10.694645+0000","last_change":"2024-04-24T01:55:18.278351+0000","last_active":"2024-04-24T02:01:10.694645+0000","last_peered":"2024-04-24T02:01:10.694645+0000","last_clean":"2024-04-24T02:01:10.694645+0000","last_became_active":"2024-04-24T01:55:18.273668+0000","last_became_peered":"2024-04-24T01:55:18.273668+0000","last_unstale":"2024-04-24T02:01:10.694645+0000","last_undegraded":"2024-04-24T02:01:10.694645+0000","last_fullsized":"2024-04-24T02:01:10.694645+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"342'369","reported_seq":1391,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:10.689816+0000","last_change":"2024-04-24T02:01:00.639502+0000","last_active":"2024-04-24T02:01:10.689816+0000","last_peered":"2024-04-24T02:01:10.689816+0000","last_clean":"2024-04-24T02:01:10.689816+0000","last_became_active":"2024-04-24T01:55:30.363123+0000","last_became_peered":"2024-04-24T01:55:30.363123+0000","last_unstale":"2024-04-24T02:01:10.689816+0000","last_undegraded":"2024-04-24T02:01:10.689816+0000","last_fullsized":"2024-04-24T02:01:10.689816+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":369,"log_dups_size":0,"ondisk_log_size":369,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":41,"snaptrim_duration":0,"stat_sum":{"num_bytes":6185240,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":249,"num_read_kb":218490,"num_write":191,"num_write_kb":50639,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":872,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:12.156807+0000","last_change":"2024-04-24T01:55:18.241117+0000","last_active":"2024-04-24T02:01:12.156807+0000","last_peered":"2024-04-24T02:01:12.156807+0000","last_clean":"2024-04-24T02:01:12.156807+0000","last_became_active":"2024-04-24T01:55:18.234313+0000","last_became_peered":"2024-04-24T01:55:18.234313+0000","last_unstale":"2024-04-24T02:01:12.156807+0000","last_undegraded":"2024-04-24T02:01:12.156807+0000","last_fullsized":"2024-04-24T02:01:12.156807+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":94257385,"num_objects":47,"num_object_clones":14,"num_object_copies":141,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":2469,"num_read_kb":1806844,"num_write":2250,"num_write_kb":609321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":211218432,"data_stored":226516047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4556,"ondisk_log_size":4556,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":1}],"osd_stats":[{"osd":3,"up_from":10,"seq":42949673319,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":99700,"kb_used_data":60396,"kb_used_omap":0,"kb_used_meta":39296,"kb_avail":94272140,"statfs":{"total":96636764160,"available":96534671360,"internally_reserved":0,"allocated":61845504,"data_stored":66195434,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":40239104},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673319,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":82876,"kb_used_data":47732,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94288964,"statfs":{"total":96636764160,"available":96551899136,"internally_reserved":0,"allocated":48877568,"data_stored":53464844,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706024,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":82884,"kb_used_data":47740,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94288956,"statfs":{"total":96636764160,"available":96551890944,"internally_reserved":0,"allocated":48885760,"data_stored":52348129,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706023,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":87788,"kb_used_data":52644,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94284052,"statfs":{"total":96636764160,"available":96546869248,"internally_reserved":0,"allocated":53907456,"data_stored":57205636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":48295936,"data_stored":51526038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53379072,"data_stored":56973913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":48287744,"data_stored":52642753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61255680,"data_stored":65373343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T02:01:16.545 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T02:01:16.650 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:01:16.645+0000 7feacf46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:01:16.654 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:01:16.653+0000 7feacf46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:01:16.841 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T02:01:16.841 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":351,"fsid":"e600dd1c-0bef-4ea0-91c9-2f649f75622f","created":"2024-04-24T01:55:06.017534+0000","modified":"2024-04-24T02:01:10.663781+0000","last_up_change":"2024-04-24T01:55:14.120010+0000","last_in_change":"2024-04-24T01:55:08.126530+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T01:55:14.952331+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.75,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-24T01:55:16.243536+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"unique_pool_0","create_time":"2024-04-24T01:55:28.358480+0000","flags":270369,"flags_names":"hashpspool,nopgchange,selfmanaged_snaps,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"343","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":165,"snap_epoch":343,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1,"score_stable":1,"optimal_score":1,"raw_score_acting":1,"raw_score_stable":1,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1b20709f-f1e7-4d59-8402-0d13620f747e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6800","nonce":3830851766}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6803","nonce":3830851766}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6805","nonce":3830851766}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":3830851766}]},"public_addr":"172.21.15.79:6800/3830851766","cluster_addr":"172.21.15.79:6803/3830851766","heartbeat_back_addr":"172.21.15.79:6805/3830851766","heartbeat_front_addr":"172.21.15.79:6808/3830851766","state":["exists","up"]},{"osd":1,"uuid":"54fd0eee-6d68-4c92-9ac7-6b21c87783f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6801","nonce":1823728464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":1823728464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":1823728464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6807","nonce":1823728464}]},"public_addr":"172.21.15.79:6801/1823728464","cluster_addr":"172.21.15.79:6802/1823728464","heartbeat_back_addr":"172.21.15.79:6806/1823728464","heartbeat_front_addr":"172.21.15.79:6807/1823728464","state":["exists","up"]},{"osd":2,"uuid":"35820c96-5f2a-4e2e-bc4b-714ab8e1c841","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6803","nonce":2585849279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":2585849279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":2585849279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6817","nonce":2585849279}]},"public_addr":"172.21.15.103:6803/2585849279","cluster_addr":"172.21.15.103:6804/2585849279","heartbeat_back_addr":"172.21.15.103:6816/2585849279","heartbeat_front_addr":"172.21.15.103:6817/2585849279","state":["exists","up"]},{"osd":3,"uuid":"da975ea1-eeb6-4619-b73f-786b46dd8fe6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6809","nonce":659350107}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":659350107}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":659350107}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6811","nonce":659350107}]},"public_addr":"172.21.15.103:6809/659350107","cluster_addr":"172.21.15.103:6808/659350107","heartbeat_back_addr":"172.21.15.103:6812/659350107","heartbeat_front_addr":"172.21.15.103:6811/659350107","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":158},{"begin":161,"length":1},{"begin":163,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T02:01:17.856 INFO:tasks.ceph:Scrubbing osd.0 2024-04-24T02:01:17.856 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-04-24T02:01:17.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:01:17.971+0000 7f2aee19d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:01:17.983 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:01:17.983+0000 7f2aee19d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:01:17.997 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-04-24T02:01:17.997 INFO:teuthology.orchestra.run.smithi079.stdout: "success": "" 2024-04-24T02:01:17.997 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-04-24T02:01:18.009 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-04-24T02:01:18.118 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:01:18.114+0000 7fc178ced640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:01:18.122 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:01:18.122+0000 7fc178ced640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:01:18.322 INFO:teuthology.orchestra.run.smithi079.stderr:instructed osd(s) 0 to deep-scrub 2024-04-24T02:01:18.344 INFO:tasks.ceph:Scrubbing osd.1 2024-04-24T02:01:18.344 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-04-24T02:01:18.471 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:01:18.463+0000 7f0ba679d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:01:18.475 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:01:18.475+0000 7f0ba679d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:01:18.488 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-04-24T02:01:18.489 INFO:teuthology.orchestra.run.smithi079.stdout: "success": "" 2024-04-24T02:01:18.489 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-04-24T02:01:18.500 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-04-24T02:01:18.608 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:01:18.603+0000 7f69a251d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:01:18.611 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:01:18.611+0000 7f69a251d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:01:18.801 INFO:teuthology.orchestra.run.smithi079.stderr:instructed osd(s) 1 to deep-scrub 2024-04-24T02:01:18.825 INFO:tasks.ceph:Scrubbing osd.2 2024-04-24T02:01:18.825 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-04-24T02:01:18.962 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:01:18.954+0000 7f6ac039d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:01:18.965 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:01:18.966+0000 7f6ac039d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:01:18.981 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-04-24T02:01:18.981 INFO:teuthology.orchestra.run.smithi079.stdout: "success": "" 2024-04-24T02:01:18.981 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-04-24T02:01:18.992 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-04-24T02:01:19.099 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:01:19.094+0000 7fee078ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:01:19.103 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:01:19.103+0000 7fee078ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:01:19.297 INFO:teuthology.orchestra.run.smithi079.stderr:instructed osd(s) 2 to deep-scrub 2024-04-24T02:01:19.312 INFO:tasks.ceph:Scrubbing osd.3 2024-04-24T02:01:19.312 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-04-24T02:01:19.437 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:01:19.429+0000 7f49f5b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:01:19.440 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:01:19.441+0000 7f49f5b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:01:19.456 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-04-24T02:01:19.456 INFO:teuthology.orchestra.run.smithi079.stdout: "success": "" 2024-04-24T02:01:19.457 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-04-24T02:01:19.467 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-04-24T02:01:19.574 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:01:19.570+0000 7f419b46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:01:19.578 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:01:19.578+0000 7f419b46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:01:19.764 INFO:teuthology.orchestra.run.smithi079.stderr:instructed osd(s) 3 to deep-scrub 2024-04-24T02:01:19.778 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T02:01:19.917 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:01:19.914+0000 7fd27d0ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:01:19.921 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:01:19.921+0000 7fd27d0ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:01:20.111 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T02:01:20.111 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-24T02:01:20.123 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":525,"stamp":"2024-04-24T02:01:19.016388+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":94847753,"num_objects":49,"num_object_clones":14,"num_object_copies":147,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":2572,"num_read_kb":1806935,"num_write":2387,"num_write_kb":611215,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4653,"ondisk_log_size":4653,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":353248,"kb_used_data":208512,"kb_used_omap":0,"kb_used_meta":144704,"kb_avail":377134112,"statfs":{"total":386547056640,"available":386185330688,"internally_reserved":0,"allocated":213516288,"data_stored":229214043,"data_compressed":89889,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":148176896},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.351539"},"pg_stats":[{"pgid":"3.f","version":"342'255","reported_seq":1319,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:12.191911+0000","last_change":"2024-04-24T02:01:00.252850+0000","last_active":"2024-04-24T02:01:12.191911+0000","last_peered":"2024-04-24T02:01:12.191911+0000","last_clean":"2024-04-24T02:01:12.191911+0000","last_became_active":"2024-04-24T01:55:30.386726+0000","last_became_peered":"2024-04-24T01:55:30.386726+0000","last_unstale":"2024-04-24T02:01:12.191911+0000","last_undegraded":"2024-04-24T02:01:12.191911+0000","last_fullsized":"2024-04-24T02:01:12.191911+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":255,"log_dups_size":0,"ondisk_log_size":255,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":23,"snaptrim_duration":0,"stat_sum":{"num_bytes":5512960,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":185,"num_read_kb":165160,"num_write":116,"num_write_kb":33460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"326'374","reported_seq":1370,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:10.682824+0000","last_change":"2024-04-24T02:01:00.196927+0000","last_active":"2024-04-24T02:01:10.682824+0000","last_peered":"2024-04-24T02:01:10.682824+0000","last_clean":"2024-04-24T02:01:10.682824+0000","last_became_active":"2024-04-24T01:55:30.321866+0000","last_became_peered":"2024-04-24T01:55:30.321866+0000","last_unstale":"2024-04-24T02:01:10.682824+0000","last_undegraded":"2024-04-24T02:01:10.682824+0000","last_fullsized":"2024-04-24T02:01:10.682824+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":374,"log_dups_size":0,"ondisk_log_size":374,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":36,"snaptrim_duration":0,"stat_sum":{"num_bytes":2823708,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":203,"num_read_kb":136089,"num_write":185,"num_write_kb":49191,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"342'235","reported_seq":1251,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:12.147307+0000","last_change":"2024-04-24T02:01:00.191968+0000","last_active":"2024-04-24T02:01:12.147307+0000","last_peered":"2024-04-24T02:01:12.147307+0000","last_clean":"2024-04-24T02:01:12.147307+0000","last_became_active":"2024-04-24T01:55:30.320186+0000","last_became_peered":"2024-04-24T01:55:30.320186+0000","last_unstale":"2024-04-24T02:01:12.147307+0000","last_undegraded":"2024-04-24T02:01:12.147307+0000","last_fullsized":"2024-04-24T02:01:12.147307+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":235,"log_dups_size":0,"ondisk_log_size":235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":24,"snaptrim_duration":0,"stat_sum":{"num_bytes":6017104,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":139,"num_read_kb":94253,"num_write":128,"num_write_kb":35473,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"342'309","reported_seq":1347,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:12.192818+0000","last_change":"2024-04-24T02:01:00.248214+0000","last_active":"2024-04-24T02:01:12.192818+0000","last_peered":"2024-04-24T02:01:12.192818+0000","last_clean":"2024-04-24T02:01:12.192818+0000","last_became_active":"2024-04-24T01:55:30.288081+0000","last_became_peered":"2024-04-24T01:55:30.288081+0000","last_unstale":"2024-04-24T02:01:12.192818+0000","last_undegraded":"2024-04-24T02:01:12.192818+0000","last_fullsized":"2024-04-24T02:01:12.192818+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":309,"log_dups_size":0,"ondisk_log_size":309,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":30,"snaptrim_duration":0,"stat_sum":{"num_bytes":9832458,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":161,"num_read_kb":116003,"num_write":154,"num_write_kb":41482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"336'138","reported_seq":1185,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:13.184844+0000","last_change":"2024-04-24T02:00:59.336871+0000","last_active":"2024-04-24T02:01:13.184844+0000","last_peered":"2024-04-24T02:01:13.184844+0000","last_clean":"2024-04-24T02:01:13.184844+0000","last_became_active":"2024-04-24T01:55:30.333815+0000","last_became_peered":"2024-04-24T01:55:30.333815+0000","last_unstale":"2024-04-24T02:01:13.184844+0000","last_undegraded":"2024-04-24T02:01:13.184844+0000","last_fullsized":"2024-04-24T02:01:13.184844+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":17,"snaptrim_duration":0,"stat_sum":{"num_bytes":2857322,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":82,"num_read_kb":52358,"num_write":76,"num_write_kb":20051,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"342'638","reported_seq":1608,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:12.193652+0000","last_change":"2024-04-24T02:01:00.250311+0000","last_active":"2024-04-24T02:01:12.193652+0000","last_peered":"2024-04-24T02:01:12.193652+0000","last_clean":"2024-04-24T02:01:12.193652+0000","last_became_active":"2024-04-24T01:55:30.315075+0000","last_became_peered":"2024-04-24T01:55:30.315075+0000","last_unstale":"2024-04-24T02:01:12.193652+0000","last_undegraded":"2024-04-24T02:01:12.193652+0000","last_fullsized":"2024-04-24T02:01:12.193652+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":638,"log_dups_size":0,"ondisk_log_size":638,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":65,"snaptrim_duration":0,"stat_sum":{"num_bytes":8807264,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":282,"num_read_kb":182263,"num_write":282,"num_write_kb":72883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"342'150","reported_seq":1198,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:12.200097+0000","last_change":"2024-04-24T02:01:00.239736+0000","last_active":"2024-04-24T02:01:12.200097+0000","last_peered":"2024-04-24T02:01:12.200097+0000","last_clean":"2024-04-24T02:01:12.200097+0000","last_became_active":"2024-04-24T01:55:30.350300+0000","last_became_peered":"2024-04-24T01:55:30.350300+0000","last_unstale":"2024-04-24T02:01:12.200097+0000","last_undegraded":"2024-04-24T02:01:12.200097+0000","last_fullsized":"2024-04-24T02:01:12.200097+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":150,"log_dups_size":0,"ondisk_log_size":150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":13,"snaptrim_duration":0,"stat_sum":{"num_bytes":3243883,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":66,"num_read_kb":40460,"num_write":87,"num_write_kb":22401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"264'34","reported_seq":1059,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:12.155738+0000","last_change":"2024-04-24T02:01:00.170396+0000","last_active":"2024-04-24T02:01:12.155738+0000","last_peered":"2024-04-24T02:01:12.155738+0000","last_clean":"2024-04-24T02:01:12.155738+0000","last_became_active":"2024-04-24T01:55:30.376813+0000","last_became_peered":"2024-04-24T01:55:30.376813+0000","last_unstale":"2024-04-24T02:01:12.155738+0000","last_undegraded":"2024-04-24T02:01:12.155738+0000","last_fullsized":"2024-04-24T02:01:12.155738+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0,"stat_sum":{"num_bytes":3025359,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":21,"num_read_kb":15411,"num_write":27,"num_write_kb":6802,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'97","reported_seq":1031,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:13.169480+0000","last_change":"2024-04-24T01:55:16.196470+0000","last_active":"2024-04-24T02:01:13.169480+0000","last_peered":"2024-04-24T02:01:13.169480+0000","last_clean":"2024-04-24T02:01:13.169480+0000","last_became_active":"2024-04-24T01:55:16.190464+0000","last_became_peered":"2024-04-24T01:55:16.190464+0000","last_unstale":"2024-04-24T02:01:13.169480+0000","last_undegraded":"2024-04-24T02:01:13.169480+0000","last_fullsized":"2024-04-24T02:01:13.169480+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:15.127115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:15.127115+0000","last_clean_scrub_stamp":"2024-04-24T01:55:15.127115+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":872,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:12.153789+0000","last_change":"2024-04-24T01:55:18.236148+0000","last_active":"2024-04-24T02:01:12.153789+0000","last_peered":"2024-04-24T02:01:12.153789+0000","last_clean":"2024-04-24T02:01:12.153789+0000","last_became_active":"2024-04-24T01:55:18.230154+0000","last_became_peered":"2024-04-24T01:55:18.230154+0000","last_unstale":"2024-04-24T02:01:12.153789+0000","last_undegraded":"2024-04-24T02:01:12.153789+0000","last_fullsized":"2024-04-24T02:01:12.153789+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"342'462","reported_seq":1458,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:12.153249+0000","last_change":"2024-04-24T02:01:00.337592+0000","last_active":"2024-04-24T02:01:12.153249+0000","last_peered":"2024-04-24T02:01:12.153249+0000","last_clean":"2024-04-24T02:01:12.153249+0000","last_became_active":"2024-04-24T01:55:30.316222+0000","last_became_peered":"2024-04-24T01:55:30.316222+0000","last_unstale":"2024-04-24T02:01:12.153249+0000","last_undegraded":"2024-04-24T02:01:12.153249+0000","last_fullsized":"2024-04-24T02:01:12.153249+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":462,"log_dups_size":0,"ondisk_log_size":462,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":39,"snaptrim_duration":0,"stat_sum":{"num_bytes":3025392,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":231,"num_read_kb":163233,"num_write":236,"num_write_kb":65892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":908,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:12.201364+0000","last_change":"2024-04-24T01:55:18.265905+0000","last_active":"2024-04-24T02:01:12.201364+0000","last_peered":"2024-04-24T02:01:12.201364+0000","last_clean":"2024-04-24T02:01:12.201364+0000","last_became_active":"2024-04-24T01:55:18.257020+0000","last_became_peered":"2024-04-24T01:55:18.257020+0000","last_unstale":"2024-04-24T02:01:12.201364+0000","last_undegraded":"2024-04-24T02:01:12.201364+0000","last_fullsized":"2024-04-24T02:01:12.201364+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"342'358","reported_seq":1367,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:13.173644+0000","last_change":"2024-04-24T02:00:59.359790+0000","last_active":"2024-04-24T02:01:13.173644+0000","last_peered":"2024-04-24T02:01:13.173644+0000","last_clean":"2024-04-24T02:01:13.173644+0000","last_became_active":"2024-04-24T01:55:30.344129+0000","last_became_peered":"2024-04-24T01:55:30.344129+0000","last_unstale":"2024-04-24T02:01:13.173644+0000","last_undegraded":"2024-04-24T02:01:13.173644+0000","last_fullsized":"2024-04-24T02:01:13.173644+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":358,"log_dups_size":0,"ondisk_log_size":358,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":31,"snaptrim_duration":0,"stat_sum":{"num_bytes":10420703,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":155,"num_read_kb":92133,"num_write":178,"num_write_kb":50302,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":872,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:12.146036+0000","last_change":"2024-04-24T01:55:18.215251+0000","last_active":"2024-04-24T02:01:12.146036+0000","last_peered":"2024-04-24T02:01:12.146036+0000","last_clean":"2024-04-24T02:01:12.146036+0000","last_became_active":"2024-04-24T01:55:18.206605+0000","last_became_peered":"2024-04-24T01:55:18.206605+0000","last_unstale":"2024-04-24T02:01:12.146036+0000","last_undegraded":"2024-04-24T02:01:12.146036+0000","last_fullsized":"2024-04-24T02:01:12.146036+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"342'208","reported_seq":1259,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:13.189650+0000","last_change":"2024-04-24T02:00:59.354665+0000","last_active":"2024-04-24T02:01:13.189650+0000","last_peered":"2024-04-24T02:01:13.189650+0000","last_clean":"2024-04-24T02:01:13.189650+0000","last_became_active":"2024-04-24T01:55:30.401366+0000","last_became_peered":"2024-04-24T01:55:30.401366+0000","last_unstale":"2024-04-24T02:01:13.189650+0000","last_undegraded":"2024-04-24T02:01:13.189650+0000","last_fullsized":"2024-04-24T02:01:13.189650+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":208,"log_dups_size":0,"ondisk_log_size":208,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":18,"snaptrim_duration":0,"stat_sum":{"num_bytes":3327918,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":153,"num_read_kb":115484,"num_write":85,"num_write_kb":21155,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":908,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:12.202080+0000","last_change":"2024-04-24T01:55:18.230893+0000","last_active":"2024-04-24T02:01:12.202080+0000","last_peered":"2024-04-24T02:01:12.202080+0000","last_clean":"2024-04-24T02:01:12.202080+0000","last_became_active":"2024-04-24T01:55:18.225188+0000","last_became_peered":"2024-04-24T01:55:18.225188+0000","last_unstale":"2024-04-24T02:01:12.202080+0000","last_undegraded":"2024-04-24T02:01:12.202080+0000","last_fullsized":"2024-04-24T02:01:12.202080+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"342'294","reported_seq":1287,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:10.678999+0000","last_change":"2024-04-24T02:01:00.246736+0000","last_active":"2024-04-24T02:01:10.678999+0000","last_peered":"2024-04-24T02:01:10.678999+0000","last_clean":"2024-04-24T02:01:10.678999+0000","last_became_active":"2024-04-24T01:55:30.343318+0000","last_became_peered":"2024-04-24T01:55:30.343318+0000","last_unstale":"2024-04-24T02:01:10.678999+0000","last_undegraded":"2024-04-24T02:01:10.678999+0000","last_fullsized":"2024-04-24T02:01:10.678999+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":294,"log_dups_size":0,"ondisk_log_size":294,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":26,"snaptrim_duration":0,"stat_sum":{"num_bytes":10118177,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":153,"num_read_kb":122486,"num_write":138,"num_write_kb":38234,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"342'437","reported_seq":1434,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:12.157896+0000","last_change":"2024-04-24T02:01:00.338179+0000","last_active":"2024-04-24T02:01:12.157896+0000","last_peered":"2024-04-24T02:01:12.157896+0000","last_clean":"2024-04-24T02:01:12.157896+0000","last_became_active":"2024-04-24T01:55:30.345308+0000","last_became_peered":"2024-04-24T01:55:30.345308+0000","last_unstale":"2024-04-24T02:01:12.157896+0000","last_undegraded":"2024-04-24T02:01:12.157896+0000","last_fullsized":"2024-04-24T02:01:12.157896+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":437,"log_dups_size":0,"ondisk_log_size":437,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":37,"snaptrim_duration":0,"stat_sum":{"num_bytes":10050949,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":237,"num_read_kb":172356,"num_write":211,"num_write_kb":56847,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":866,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:10.688889+0000","last_change":"2024-04-24T01:55:18.224238+0000","last_active":"2024-04-24T02:01:10.688889+0000","last_peered":"2024-04-24T02:01:10.688889+0000","last_clean":"2024-04-24T02:01:10.688889+0000","last_became_active":"2024-04-24T01:55:18.219131+0000","last_became_peered":"2024-04-24T01:55:18.219131+0000","last_unstale":"2024-04-24T02:01:10.688889+0000","last_undegraded":"2024-04-24T02:01:10.688889+0000","last_fullsized":"2024-04-24T02:01:10.688889+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"342'103","reported_seq":1116,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:10.693419+0000","last_change":"2024-04-24T02:01:00.249384+0000","last_active":"2024-04-24T02:01:10.693419+0000","last_peered":"2024-04-24T02:01:10.693419+0000","last_clean":"2024-04-24T02:01:10.693419+0000","last_became_active":"2024-04-24T01:55:30.319958+0000","last_became_peered":"2024-04-24T01:55:30.319958+0000","last_unstale":"2024-04-24T02:01:10.693419+0000","last_undegraded":"2024-04-24T02:01:10.693419+0000","last_fullsized":"2024-04-24T02:01:10.693419+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":8,"snaptrim_duration":0,"stat_sum":{"num_bytes":3445567,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":50,"num_read_kb":44680,"num_write":61,"num_write_kb":17461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":898,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:13.179399+0000","last_change":"2024-04-24T01:55:18.235763+0000","last_active":"2024-04-24T02:01:13.179399+0000","last_peered":"2024-04-24T02:01:13.179399+0000","last_clean":"2024-04-24T02:01:13.179399+0000","last_became_active":"2024-04-24T01:55:18.229175+0000","last_became_peered":"2024-04-24T01:55:18.229175+0000","last_unstale":"2024-04-24T02:01:13.179399+0000","last_undegraded":"2024-04-24T02:01:13.179399+0000","last_fullsized":"2024-04-24T02:01:13.179399+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"336'192","reported_seq":1228,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:13.172461+0000","last_change":"2024-04-24T02:00:59.343900+0000","last_active":"2024-04-24T02:01:13.172461+0000","last_peered":"2024-04-24T02:01:13.172461+0000","last_clean":"2024-04-24T02:01:13.172461+0000","last_became_active":"2024-04-24T01:55:30.372690+0000","last_became_peered":"2024-04-24T01:55:30.372690+0000","last_unstale":"2024-04-24T02:01:13.172461+0000","last_undegraded":"2024-04-24T02:01:13.172461+0000","last_fullsized":"2024-04-24T02:01:13.172461+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":192,"log_dups_size":0,"ondisk_log_size":192,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":18,"snaptrim_duration":0,"stat_sum":{"num_bytes":5563381,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":102,"num_read_kb":75985,"num_write":95,"num_write_kb":27048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":866,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:10.694645+0000","last_change":"2024-04-24T01:55:18.278351+0000","last_active":"2024-04-24T02:01:10.694645+0000","last_peered":"2024-04-24T02:01:10.694645+0000","last_clean":"2024-04-24T02:01:10.694645+0000","last_became_active":"2024-04-24T01:55:18.273668+0000","last_became_peered":"2024-04-24T01:55:18.273668+0000","last_unstale":"2024-04-24T02:01:10.694645+0000","last_undegraded":"2024-04-24T02:01:10.694645+0000","last_fullsized":"2024-04-24T02:01:10.694645+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"342'369","reported_seq":1391,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:10.689816+0000","last_change":"2024-04-24T02:01:00.639502+0000","last_active":"2024-04-24T02:01:10.689816+0000","last_peered":"2024-04-24T02:01:10.689816+0000","last_clean":"2024-04-24T02:01:10.689816+0000","last_became_active":"2024-04-24T01:55:30.363123+0000","last_became_peered":"2024-04-24T01:55:30.363123+0000","last_unstale":"2024-04-24T02:01:10.689816+0000","last_undegraded":"2024-04-24T02:01:10.689816+0000","last_fullsized":"2024-04-24T02:01:10.689816+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:29.168040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":369,"log_dups_size":0,"ondisk_log_size":369,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":41,"snaptrim_duration":0,"stat_sum":{"num_bytes":6185240,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":249,"num_read_kb":218490,"num_write":191,"num_write_kb":50639,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":872,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:12.156807+0000","last_change":"2024-04-24T01:55:18.241117+0000","last_active":"2024-04-24T02:01:12.156807+0000","last_peered":"2024-04-24T02:01:12.156807+0000","last_clean":"2024-04-24T02:01:12.156807+0000","last_became_active":"2024-04-24T01:55:18.234313+0000","last_became_peered":"2024-04-24T01:55:18.234313+0000","last_unstale":"2024-04-24T02:01:12.156807+0000","last_undegraded":"2024-04-24T02:01:12.156807+0000","last_fullsized":"2024-04-24T02:01:12.156807+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T01:55:17.134477+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":94257385,"num_objects":47,"num_object_clones":14,"num_object_copies":141,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":2469,"num_read_kb":1806844,"num_write":2250,"num_write_kb":609321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":211218432,"data_stored":226516047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4556,"ondisk_log_size":4556,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":1}],"osd_stats":[{"osd":3,"up_from":10,"seq":42949673319,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":99700,"kb_used_data":60396,"kb_used_omap":0,"kb_used_meta":39296,"kb_avail":94272140,"statfs":{"total":96636764160,"available":96534671360,"internally_reserved":0,"allocated":61845504,"data_stored":66195434,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":40239104},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673319,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":82876,"kb_used_data":47732,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94288964,"statfs":{"total":96636764160,"available":96551899136,"internally_reserved":0,"allocated":48877568,"data_stored":53464844,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706024,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":82884,"kb_used_data":47740,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94288956,"statfs":{"total":96636764160,"available":96551890944,"internally_reserved":0,"allocated":48885760,"data_stored":52348129,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706028,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":87788,"kb_used_data":52644,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94284052,"statfs":{"total":96636764160,"available":96546869248,"internally_reserved":0,"allocated":53907456,"data_stored":57205636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":48295936,"data_stored":51526038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53379072,"data_stored":56973913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":48287744,"data_stored":52642753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61255680,"data_stored":65373343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T02:01:20.128 INFO:tasks.ceph:pgid 3.f last_scrub_stamp 2024-04-24T01:55:29.168040+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=1, tm_min=55, tm_sec=29, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=2, tm_min=1, tm_sec=16, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T02:01:20.128 INFO:tasks.ceph:pgid 3.e last_scrub_stamp 2024-04-24T01:55:29.168040+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=1, tm_min=55, tm_sec=29, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=2, tm_min=1, tm_sec=16, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T02:01:20.128 INFO:tasks.ceph:pgid 3.d last_scrub_stamp 2024-04-24T01:55:29.168040+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=1, tm_min=55, tm_sec=29, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=2, tm_min=1, tm_sec=16, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T02:01:20.128 INFO:tasks.ceph:pgid 3.c last_scrub_stamp 2024-04-24T01:55:29.168040+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=1, tm_min=55, tm_sec=29, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=2, tm_min=1, tm_sec=16, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T02:01:20.128 INFO:tasks.ceph:pgid 3.b last_scrub_stamp 2024-04-24T01:55:29.168040+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=1, tm_min=55, tm_sec=29, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=2, tm_min=1, tm_sec=16, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T02:01:20.128 INFO:tasks.ceph:pgid 3.a last_scrub_stamp 2024-04-24T01:55:29.168040+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=1, tm_min=55, tm_sec=29, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=2, tm_min=1, tm_sec=16, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T02:01:20.128 INFO:tasks.ceph:pgid 3.9 last_scrub_stamp 2024-04-24T01:55:29.168040+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=1, tm_min=55, tm_sec=29, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=2, tm_min=1, tm_sec=16, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T02:01:20.129 INFO:tasks.ceph:pgid 3.8 last_scrub_stamp 2024-04-24T01:55:29.168040+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=1, tm_min=55, tm_sec=29, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=2, tm_min=1, tm_sec=16, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T02:01:20.129 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-04-24T01:55:15.127115+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=1, tm_min=55, tm_sec=15, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=2, tm_min=1, tm_sec=16, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T02:01:20.129 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-04-24T01:55:17.134477+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=1, tm_min=55, tm_sec=17, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=2, tm_min=1, tm_sec=16, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T02:01:20.129 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-04-24T01:55:29.168040+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=1, tm_min=55, tm_sec=29, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=2, tm_min=1, tm_sec=16, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T02:01:20.129 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-04-24T01:55:17.134477+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=1, tm_min=55, tm_sec=17, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=2, tm_min=1, tm_sec=16, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T02:01:20.129 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-04-24T01:55:29.168040+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=1, tm_min=55, tm_sec=29, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=2, tm_min=1, tm_sec=16, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T02:01:20.129 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-04-24T01:55:17.134477+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=1, tm_min=55, tm_sec=17, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=2, tm_min=1, tm_sec=16, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T02:01:20.129 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-04-24T01:55:29.168040+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=1, tm_min=55, tm_sec=29, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=2, tm_min=1, tm_sec=16, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T02:01:20.129 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-04-24T01:55:17.134477+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=1, tm_min=55, tm_sec=17, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=2, tm_min=1, tm_sec=16, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T02:01:20.130 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-04-24T01:55:29.168040+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=1, tm_min=55, tm_sec=29, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=2, tm_min=1, tm_sec=16, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T02:01:20.130 INFO:tasks.ceph:pgid 3.5 last_scrub_stamp 2024-04-24T01:55:29.168040+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=1, tm_min=55, tm_sec=29, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=2, tm_min=1, tm_sec=16, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T02:01:20.130 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-04-24T01:55:17.134477+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=1, tm_min=55, tm_sec=17, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=2, tm_min=1, tm_sec=16, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T02:01:20.130 INFO:tasks.ceph:pgid 3.4 last_scrub_stamp 2024-04-24T01:55:29.168040+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=1, tm_min=55, tm_sec=29, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=2, tm_min=1, tm_sec=16, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T02:01:20.130 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-04-24T01:55:17.134477+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=1, tm_min=55, tm_sec=17, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=2, tm_min=1, tm_sec=16, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T02:01:20.130 INFO:tasks.ceph:pgid 3.7 last_scrub_stamp 2024-04-24T01:55:29.168040+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=1, tm_min=55, tm_sec=29, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=2, tm_min=1, tm_sec=16, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T02:01:20.130 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-04-24T01:55:17.134477+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=1, tm_min=55, tm_sec=17, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=2, tm_min=1, tm_sec=16, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T02:01:20.130 INFO:tasks.ceph:pgid 3.6 last_scrub_stamp 2024-04-24T01:55:29.168040+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=1, tm_min=55, tm_sec=29, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=2, tm_min=1, tm_sec=16, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T02:01:20.131 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-04-24T01:55:17.134477+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=1, tm_min=55, tm_sec=17, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=2, tm_min=1, tm_sec=16, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T02:01:20.131 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-04-24T02:01:40.131 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T02:01:40.254 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:01:40.249+0000 7fe8aaa6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:01:40.258 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:01:40.258+0000 7fe8aaa6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:01:40.444 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-24T02:01:40.444 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-24T02:01:40.456 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":535,"stamp":"2024-04-24T02:01:39.021354+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":94847753,"num_objects":49,"num_object_clones":14,"num_object_copies":147,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":2572,"num_read_kb":1806935,"num_write":2387,"num_write_kb":611215,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4653,"ondisk_log_size":4653,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":353248,"kb_used_data":208512,"kb_used_omap":0,"kb_used_meta":144704,"kb_avail":377134112,"statfs":{"total":386547056640,"available":386185330688,"internally_reserved":0,"allocated":213516288,"data_stored":229214043,"data_compressed":89889,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":148176896},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002972"},"pg_stats":[{"pgid":"3.f","version":"342'255","reported_seq":1321,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:18.327398+0000","last_change":"2024-04-24T02:01:18.434615+0000","last_active":"2024-04-24T02:01:18.327398+0000","last_peered":"2024-04-24T02:01:18.327398+0000","last_clean":"2024-04-24T02:01:18.327398+0000","last_became_active":"2024-04-24T01:55:30.386726+0000","last_became_peered":"2024-04-24T01:55:30.386726+0000","last_unstale":"2024-04-24T02:01:18.327398+0000","last_undegraded":"2024-04-24T02:01:18.327398+0000","last_fullsized":"2024-04-24T02:01:18.327398+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"342'255","last_scrub_stamp":"2024-04-24T02:01:18.434378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T02:01:18.434378+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":255,"log_dups_size":0,"ondisk_log_size":255,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":23,"snaptrim_duration":0,"stat_sum":{"num_bytes":5512960,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":185,"num_read_kb":165160,"num_write":116,"num_write_kb":33460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"326'374","reported_seq":1372,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:19.301378+0000","last_change":"2024-04-24T02:01:19.394231+0000","last_active":"2024-04-24T02:01:19.301378+0000","last_peered":"2024-04-24T02:01:19.301378+0000","last_clean":"2024-04-24T02:01:19.301378+0000","last_became_active":"2024-04-24T01:55:30.321866+0000","last_became_peered":"2024-04-24T01:55:30.321866+0000","last_unstale":"2024-04-24T02:01:19.301378+0000","last_undegraded":"2024-04-24T02:01:19.301378+0000","last_fullsized":"2024-04-24T02:01:19.301378+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'374","last_scrub_stamp":"2024-04-24T02:01:19.393991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T02:01:19.393991+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":374,"log_dups_size":0,"ondisk_log_size":374,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":36,"snaptrim_duration":0,"stat_sum":{"num_bytes":2823708,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":203,"num_read_kb":136089,"num_write":185,"num_write_kb":49191,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"342'235","reported_seq":1253,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:18.807385+0000","last_change":"2024-04-24T02:01:18.917833+0000","last_active":"2024-04-24T02:01:18.807385+0000","last_peered":"2024-04-24T02:01:18.807385+0000","last_clean":"2024-04-24T02:01:18.807385+0000","last_became_active":"2024-04-24T01:55:30.320186+0000","last_became_peered":"2024-04-24T01:55:30.320186+0000","last_unstale":"2024-04-24T02:01:18.807385+0000","last_undegraded":"2024-04-24T02:01:18.807385+0000","last_fullsized":"2024-04-24T02:01:18.807385+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"342'235","last_scrub_stamp":"2024-04-24T02:01:18.917638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T02:01:18.917638+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":235,"log_dups_size":0,"ondisk_log_size":235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":24,"snaptrim_duration":0,"stat_sum":{"num_bytes":6017104,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":139,"num_read_kb":94253,"num_write":128,"num_write_kb":35473,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"342'309","reported_seq":1349,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:18.326501+0000","last_change":"2024-04-24T02:01:18.452051+0000","last_active":"2024-04-24T02:01:18.326501+0000","last_peered":"2024-04-24T02:01:18.326501+0000","last_clean":"2024-04-24T02:01:18.326501+0000","last_became_active":"2024-04-24T01:55:30.288081+0000","last_became_peered":"2024-04-24T01:55:30.288081+0000","last_unstale":"2024-04-24T02:01:18.326501+0000","last_undegraded":"2024-04-24T02:01:18.326501+0000","last_fullsized":"2024-04-24T02:01:18.326501+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"342'309","last_scrub_stamp":"2024-04-24T02:01:18.451838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T02:01:18.451838+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":309,"log_dups_size":0,"ondisk_log_size":309,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":30,"snaptrim_duration":0,"stat_sum":{"num_bytes":9832458,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":161,"num_read_kb":116003,"num_write":154,"num_write_kb":41482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"336'138","reported_seq":1187,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:19.770990+0000","last_change":"2024-04-24T02:01:19.830033+0000","last_active":"2024-04-24T02:01:19.770990+0000","last_peered":"2024-04-24T02:01:19.770990+0000","last_clean":"2024-04-24T02:01:19.770990+0000","last_became_active":"2024-04-24T01:55:30.333815+0000","last_became_peered":"2024-04-24T01:55:30.333815+0000","last_unstale":"2024-04-24T02:01:19.770990+0000","last_undegraded":"2024-04-24T02:01:19.770990+0000","last_fullsized":"2024-04-24T02:01:19.770990+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"336'138","last_scrub_stamp":"2024-04-24T02:01:19.829687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T02:01:19.829687+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":17,"snaptrim_duration":0,"stat_sum":{"num_bytes":2857322,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":82,"num_read_kb":52358,"num_write":76,"num_write_kb":20051,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"342'638","reported_seq":1610,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:18.325523+0000","last_change":"2024-04-24T02:01:18.464439+0000","last_active":"2024-04-24T02:01:18.325523+0000","last_peered":"2024-04-24T02:01:18.325523+0000","last_clean":"2024-04-24T02:01:18.325523+0000","last_became_active":"2024-04-24T01:55:30.315075+0000","last_became_peered":"2024-04-24T01:55:30.315075+0000","last_unstale":"2024-04-24T02:01:18.325523+0000","last_undegraded":"2024-04-24T02:01:18.325523+0000","last_fullsized":"2024-04-24T02:01:18.325523+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"342'638","last_scrub_stamp":"2024-04-24T02:01:18.464234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T02:01:18.464234+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":638,"log_dups_size":0,"ondisk_log_size":638,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":65,"snaptrim_duration":0,"stat_sum":{"num_bytes":8807264,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":282,"num_read_kb":182263,"num_write":282,"num_write_kb":72883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"342'150","reported_seq":1200,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:18.326915+0000","last_change":"2024-04-24T02:01:18.447603+0000","last_active":"2024-04-24T02:01:18.326915+0000","last_peered":"2024-04-24T02:01:18.326915+0000","last_clean":"2024-04-24T02:01:18.326915+0000","last_became_active":"2024-04-24T01:55:30.350300+0000","last_became_peered":"2024-04-24T01:55:30.350300+0000","last_unstale":"2024-04-24T02:01:18.326915+0000","last_undegraded":"2024-04-24T02:01:18.326915+0000","last_fullsized":"2024-04-24T02:01:18.326915+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"342'150","last_scrub_stamp":"2024-04-24T02:01:18.447230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T02:01:18.447230+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":150,"log_dups_size":0,"ondisk_log_size":150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":13,"snaptrim_duration":0,"stat_sum":{"num_bytes":3243883,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":66,"num_read_kb":40460,"num_write":87,"num_write_kb":22401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"264'34","reported_seq":1061,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:18.809851+0000","last_change":"2024-04-24T02:01:18.876749+0000","last_active":"2024-04-24T02:01:18.809851+0000","last_peered":"2024-04-24T02:01:18.809851+0000","last_clean":"2024-04-24T02:01:18.809851+0000","last_became_active":"2024-04-24T01:55:30.376813+0000","last_became_peered":"2024-04-24T01:55:30.376813+0000","last_unstale":"2024-04-24T02:01:18.809851+0000","last_undegraded":"2024-04-24T02:01:18.809851+0000","last_fullsized":"2024-04-24T02:01:18.809851+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"264'34","last_scrub_stamp":"2024-04-24T02:01:18.876372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T02:01:18.876372+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0,"stat_sum":{"num_bytes":3025359,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":21,"num_read_kb":15411,"num_write":27,"num_write_kb":6802,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'97","reported_seq":1033,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:19.766457+0000","last_change":"2024-04-24T02:01:19.827314+0000","last_active":"2024-04-24T02:01:19.766457+0000","last_peered":"2024-04-24T02:01:19.766457+0000","last_clean":"2024-04-24T02:01:19.766457+0000","last_became_active":"2024-04-24T01:55:16.190464+0000","last_became_peered":"2024-04-24T01:55:16.190464+0000","last_unstale":"2024-04-24T02:01:19.766457+0000","last_undegraded":"2024-04-24T02:01:19.766457+0000","last_fullsized":"2024-04-24T02:01:19.766457+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"13'97","last_scrub_stamp":"2024-04-24T02:01:19.827089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T02:01:19.827089+0000","last_clean_scrub_stamp":"2024-04-24T01:55:15.127115+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":874,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:18.805884+0000","last_change":"2024-04-24T02:01:18.856874+0000","last_active":"2024-04-24T02:01:18.805884+0000","last_peered":"2024-04-24T02:01:18.805884+0000","last_clean":"2024-04-24T02:01:18.805884+0000","last_became_active":"2024-04-24T01:55:18.230154+0000","last_became_peered":"2024-04-24T01:55:18.230154+0000","last_unstale":"2024-04-24T02:01:18.805884+0000","last_undegraded":"2024-04-24T02:01:18.805884+0000","last_fullsized":"2024-04-24T02:01:18.805884+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T02:01:18.856593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T02:01:18.856593+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"342'462","reported_seq":1460,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:18.808836+0000","last_change":"2024-04-24T02:01:18.911065+0000","last_active":"2024-04-24T02:01:18.808836+0000","last_peered":"2024-04-24T02:01:18.808836+0000","last_clean":"2024-04-24T02:01:18.808836+0000","last_became_active":"2024-04-24T01:55:30.316222+0000","last_became_peered":"2024-04-24T01:55:30.316222+0000","last_unstale":"2024-04-24T02:01:18.808836+0000","last_undegraded":"2024-04-24T02:01:18.808836+0000","last_fullsized":"2024-04-24T02:01:18.808836+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"342'462","last_scrub_stamp":"2024-04-24T02:01:18.910841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T02:01:18.910841+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":462,"log_dups_size":0,"ondisk_log_size":462,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":39,"snaptrim_duration":0,"stat_sum":{"num_bytes":3025392,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":231,"num_read_kb":163233,"num_write":236,"num_write_kb":65892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":910,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:18.325603+0000","last_change":"2024-04-24T02:01:18.359873+0000","last_active":"2024-04-24T02:01:18.325603+0000","last_peered":"2024-04-24T02:01:18.325603+0000","last_clean":"2024-04-24T02:01:18.325603+0000","last_became_active":"2024-04-24T01:55:18.257020+0000","last_became_peered":"2024-04-24T01:55:18.257020+0000","last_unstale":"2024-04-24T02:01:18.325603+0000","last_undegraded":"2024-04-24T02:01:18.325603+0000","last_fullsized":"2024-04-24T02:01:18.325603+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T02:01:18.359694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T02:01:18.359694+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"342'358","reported_seq":1369,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:19.768482+0000","last_change":"2024-04-24T02:01:19.897184+0000","last_active":"2024-04-24T02:01:19.768482+0000","last_peered":"2024-04-24T02:01:19.768482+0000","last_clean":"2024-04-24T02:01:19.768482+0000","last_became_active":"2024-04-24T01:55:30.344129+0000","last_became_peered":"2024-04-24T01:55:30.344129+0000","last_unstale":"2024-04-24T02:01:19.768482+0000","last_undegraded":"2024-04-24T02:01:19.768482+0000","last_fullsized":"2024-04-24T02:01:19.768482+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"342'358","last_scrub_stamp":"2024-04-24T02:01:19.896978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T02:01:19.896978+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":358,"log_dups_size":0,"ondisk_log_size":358,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":31,"snaptrim_duration":0,"stat_sum":{"num_bytes":10420703,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":155,"num_read_kb":92133,"num_write":178,"num_write_kb":50302,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":874,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:18.804810+0000","last_change":"2024-04-24T02:01:18.849369+0000","last_active":"2024-04-24T02:01:18.804810+0000","last_peered":"2024-04-24T02:01:18.804810+0000","last_clean":"2024-04-24T02:01:18.804810+0000","last_became_active":"2024-04-24T01:55:18.206605+0000","last_became_peered":"2024-04-24T01:55:18.206605+0000","last_unstale":"2024-04-24T02:01:18.804810+0000","last_undegraded":"2024-04-24T02:01:18.804810+0000","last_fullsized":"2024-04-24T02:01:18.804810+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T02:01:18.849064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T02:01:18.849064+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"342'208","reported_seq":1261,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:19.768535+0000","last_change":"2024-04-24T02:01:19.837872+0000","last_active":"2024-04-24T02:01:19.768535+0000","last_peered":"2024-04-24T02:01:19.768535+0000","last_clean":"2024-04-24T02:01:19.768535+0000","last_became_active":"2024-04-24T01:55:30.401366+0000","last_became_peered":"2024-04-24T01:55:30.401366+0000","last_unstale":"2024-04-24T02:01:19.768535+0000","last_undegraded":"2024-04-24T02:01:19.768535+0000","last_fullsized":"2024-04-24T02:01:19.768535+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"342'208","last_scrub_stamp":"2024-04-24T02:01:19.837661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T02:01:19.837661+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":208,"log_dups_size":0,"ondisk_log_size":208,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":18,"snaptrim_duration":0,"stat_sum":{"num_bytes":3327918,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":153,"num_read_kb":115484,"num_write":85,"num_write_kb":21155,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":910,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:18.325799+0000","last_change":"2024-04-24T02:01:18.362529+0000","last_active":"2024-04-24T02:01:18.325799+0000","last_peered":"2024-04-24T02:01:18.325799+0000","last_clean":"2024-04-24T02:01:18.325799+0000","last_became_active":"2024-04-24T01:55:18.225188+0000","last_became_peered":"2024-04-24T01:55:18.225188+0000","last_unstale":"2024-04-24T02:01:18.325799+0000","last_undegraded":"2024-04-24T02:01:18.325799+0000","last_fullsized":"2024-04-24T02:01:18.325799+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T02:01:18.362339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T02:01:18.362339+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"342'294","reported_seq":1289,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:19.300045+0000","last_change":"2024-04-24T02:01:19.418830+0000","last_active":"2024-04-24T02:01:19.300045+0000","last_peered":"2024-04-24T02:01:19.300045+0000","last_clean":"2024-04-24T02:01:19.300045+0000","last_became_active":"2024-04-24T01:55:30.343318+0000","last_became_peered":"2024-04-24T01:55:30.343318+0000","last_unstale":"2024-04-24T02:01:19.300045+0000","last_undegraded":"2024-04-24T02:01:19.300045+0000","last_fullsized":"2024-04-24T02:01:19.300045+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"342'294","last_scrub_stamp":"2024-04-24T02:01:19.418465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T02:01:19.418465+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":294,"log_dups_size":0,"ondisk_log_size":294,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":26,"snaptrim_duration":0,"stat_sum":{"num_bytes":10118177,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":153,"num_read_kb":122486,"num_write":138,"num_write_kb":38234,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"342'437","reported_seq":1436,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:18.808140+0000","last_change":"2024-04-24T02:01:18.942887+0000","last_active":"2024-04-24T02:01:18.808140+0000","last_peered":"2024-04-24T02:01:18.808140+0000","last_clean":"2024-04-24T02:01:18.808140+0000","last_became_active":"2024-04-24T01:55:30.345308+0000","last_became_peered":"2024-04-24T01:55:30.345308+0000","last_unstale":"2024-04-24T02:01:18.808140+0000","last_undegraded":"2024-04-24T02:01:18.808140+0000","last_fullsized":"2024-04-24T02:01:18.808140+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"342'437","last_scrub_stamp":"2024-04-24T02:01:18.942657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T02:01:18.942657+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":437,"log_dups_size":0,"ondisk_log_size":437,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":37,"snaptrim_duration":0,"stat_sum":{"num_bytes":10050949,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":237,"num_read_kb":172356,"num_write":211,"num_write_kb":56847,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":868,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:19.299723+0000","last_change":"2024-04-24T02:01:19.312251+0000","last_active":"2024-04-24T02:01:19.299723+0000","last_peered":"2024-04-24T02:01:19.299723+0000","last_clean":"2024-04-24T02:01:19.299723+0000","last_became_active":"2024-04-24T01:55:18.219131+0000","last_became_peered":"2024-04-24T01:55:18.219131+0000","last_unstale":"2024-04-24T02:01:19.299723+0000","last_undegraded":"2024-04-24T02:01:19.299723+0000","last_fullsized":"2024-04-24T02:01:19.299723+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T02:01:19.312040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T02:01:19.312040+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"342'103","reported_seq":1118,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:19.302118+0000","last_change":"2024-04-24T02:01:19.417962+0000","last_active":"2024-04-24T02:01:19.302118+0000","last_peered":"2024-04-24T02:01:19.302118+0000","last_clean":"2024-04-24T02:01:19.302118+0000","last_became_active":"2024-04-24T01:55:30.319958+0000","last_became_peered":"2024-04-24T01:55:30.319958+0000","last_unstale":"2024-04-24T02:01:19.302118+0000","last_undegraded":"2024-04-24T02:01:19.302118+0000","last_fullsized":"2024-04-24T02:01:19.302118+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"342'103","last_scrub_stamp":"2024-04-24T02:01:19.417618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T02:01:19.417618+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":8,"snaptrim_duration":0,"stat_sum":{"num_bytes":3445567,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":50,"num_read_kb":44680,"num_write":61,"num_write_kb":17461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":900,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:19.768516+0000","last_change":"2024-04-24T02:01:19.805146+0000","last_active":"2024-04-24T02:01:19.768516+0000","last_peered":"2024-04-24T02:01:19.768516+0000","last_clean":"2024-04-24T02:01:19.768516+0000","last_became_active":"2024-04-24T01:55:18.229175+0000","last_became_peered":"2024-04-24T01:55:18.229175+0000","last_unstale":"2024-04-24T02:01:19.768516+0000","last_undegraded":"2024-04-24T02:01:19.768516+0000","last_fullsized":"2024-04-24T02:01:19.768516+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T02:01:19.804850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T02:01:19.804850+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"336'192","reported_seq":1230,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:19.769887+0000","last_change":"2024-04-24T02:01:19.851491+0000","last_active":"2024-04-24T02:01:19.769887+0000","last_peered":"2024-04-24T02:01:19.769887+0000","last_clean":"2024-04-24T02:01:19.769887+0000","last_became_active":"2024-04-24T01:55:30.372690+0000","last_became_peered":"2024-04-24T01:55:30.372690+0000","last_unstale":"2024-04-24T02:01:19.769887+0000","last_undegraded":"2024-04-24T02:01:19.769887+0000","last_fullsized":"2024-04-24T02:01:19.769887+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"336'192","last_scrub_stamp":"2024-04-24T02:01:19.851305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T02:01:19.851305+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":192,"log_dups_size":0,"ondisk_log_size":192,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":18,"snaptrim_duration":0,"stat_sum":{"num_bytes":5563381,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":102,"num_read_kb":75985,"num_write":95,"num_write_kb":27048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":868,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:19.301351+0000","last_change":"2024-04-24T02:01:19.321250+0000","last_active":"2024-04-24T02:01:19.301351+0000","last_peered":"2024-04-24T02:01:19.301351+0000","last_clean":"2024-04-24T02:01:19.301351+0000","last_became_active":"2024-04-24T01:55:18.273668+0000","last_became_peered":"2024-04-24T01:55:18.273668+0000","last_unstale":"2024-04-24T02:01:19.301351+0000","last_undegraded":"2024-04-24T02:01:19.301351+0000","last_fullsized":"2024-04-24T02:01:19.301351+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T02:01:19.320942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T02:01:19.320942+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"342'369","reported_seq":1393,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:19.303661+0000","last_change":"2024-04-24T02:01:19.432070+0000","last_active":"2024-04-24T02:01:19.303661+0000","last_peered":"2024-04-24T02:01:19.303661+0000","last_clean":"2024-04-24T02:01:19.303661+0000","last_became_active":"2024-04-24T01:55:30.363123+0000","last_became_peered":"2024-04-24T01:55:30.363123+0000","last_unstale":"2024-04-24T02:01:19.303661+0000","last_undegraded":"2024-04-24T02:01:19.303661+0000","last_fullsized":"2024-04-24T02:01:19.303661+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"342'369","last_scrub_stamp":"2024-04-24T02:01:19.431817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T02:01:19.431817+0000","last_clean_scrub_stamp":"2024-04-24T01:55:29.168040+0000","objects_scrubbed":0,"log_size":369,"log_dups_size":0,"ondisk_log_size":369,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":41,"snaptrim_duration":0,"stat_sum":{"num_bytes":6185240,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":249,"num_read_kb":218490,"num_write":191,"num_write_kb":50639,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":874,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-24T02:01:18.807253+0000","last_change":"2024-04-24T02:01:18.867744+0000","last_active":"2024-04-24T02:01:18.807253+0000","last_peered":"2024-04-24T02:01:18.807253+0000","last_clean":"2024-04-24T02:01:18.807253+0000","last_became_active":"2024-04-24T01:55:18.234313+0000","last_became_peered":"2024-04-24T01:55:18.234313+0000","last_unstale":"2024-04-24T02:01:18.807253+0000","last_undegraded":"2024-04-24T02:01:18.807253+0000","last_fullsized":"2024-04-24T02:01:18.807253+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T02:01:18.867437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T02:01:18.867437+0000","last_clean_scrub_stamp":"2024-04-24T01:55:17.134477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":94257385,"num_objects":47,"num_object_clones":14,"num_object_copies":141,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":2469,"num_read_kb":1806844,"num_write":2250,"num_write_kb":609321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":211218432,"data_stored":226516047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4556,"ondisk_log_size":4556,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":1}],"osd_stats":[{"osd":3,"up_from":10,"seq":42949673339,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":99700,"kb_used_data":60396,"kb_used_omap":0,"kb_used_meta":39296,"kb_avail":94272140,"statfs":{"total":96636764160,"available":96534671360,"internally_reserved":0,"allocated":61845504,"data_stored":66195434,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":40239104},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673339,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":82876,"kb_used_data":47732,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94288964,"statfs":{"total":96636764160,"available":96551899136,"internally_reserved":0,"allocated":48877568,"data_stored":53464844,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706044,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":82884,"kb_used_data":47740,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94288956,"statfs":{"total":96636764160,"available":96551890944,"internally_reserved":0,"allocated":48885760,"data_stored":52348129,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706048,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":87788,"kb_used_data":52644,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94284052,"statfs":{"total":96636764160,"available":96546869248,"internally_reserved":0,"allocated":53907456,"data_stored":57205636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29963,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":48295936,"data_stored":51526038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53379072,"data_stored":56973913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":48287744,"data_stored":52642753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61255680,"data_stored":65373343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T02:01:40.459 DEBUG:teuthology.orchestra.run.smithi079:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-04-24T02:01:40.565 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:01:40.557+0000 7f4f8e820640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:01:40.569 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-24T02:01:40.568+0000 7f4f8e820640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-24T02:01:40.773 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_424d8c4389b684b0a4d72510e1b4d398d8e6c8f4/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_424d8c4389b684b0a4d72510e1b4d398d8e6c8f4/qa/tasks/rados.py", line 308, in task running.get() File "src/gevent/greenlet.py", line 831, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/gevent/_compat.py", line 49, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_ceph-c_424d8c4389b684b0a4d72510e1b4d398d8e6c8f4/qa/tasks/rados.py", line 289, in thread run.wait(tests.values()) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 179, in _raise_for_status raise CommandCrashedError(command=self.command) teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 0 --op copy_from 0 --op write_excl 50 --pool unique_pool_0' 2024-04-24T02:01:40.774 INFO:teuthology.misc:Shutting down mds daemons... 2024-04-24T02:01:40.774 INFO:teuthology.misc:Shutting down osd daemons... 2024-04-24T02:01:40.775 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-04-24T02:01:40.775 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T02:01:40.850 INFO:tasks.ceph.osd.0:Stopped 2024-04-24T02:01:40.851 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-04-24T02:01:40.851 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T02:01:40.931 INFO:tasks.ceph.osd.1:Stopped 2024-04-24T02:01:40.931 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-04-24T02:01:40.931 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T02:01:41.009 INFO:tasks.ceph.osd.2:Stopped 2024-04-24T02:01:41.009 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-04-24T02:01:41.009 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T02:01:41.084 INFO:tasks.ceph.osd.3:Stopped 2024-04-24T02:01:41.084 INFO:teuthology.misc:Shutting down mgr daemons... 2024-04-24T02:01:41.084 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-04-24T02:01:41.084 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T02:01:41.111 INFO:tasks.ceph.mgr.x:Stopped 2024-04-24T02:01:41.111 INFO:teuthology.misc:Shutting down mon daemons... 2024-04-24T02:01:41.111 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-04-24T02:01:41.111 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T02:01:41.122 INFO:tasks.ceph.mon.a:Stopped 2024-04-24T02:01:41.122 INFO:tasks.ceph:Checking cluster log for badness... 2024-04-24T02:01:41.123 DEBUG:teuthology.orchestra.run.smithi079:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'mons down' | egrep -v 'mon down' | egrep -v 'out of quorum' | head -n 1 2024-04-24T02:01:41.196 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi079.front.sepia.ceph.com 2024-04-24T02:01:41.197 DEBUG:teuthology.orchestra.run.smithi079:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-04-24T02:01:41.352 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi079.front.sepia.ceph.com 2024-04-24T02:01:41.352 DEBUG:teuthology.orchestra.run.smithi079:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-04-24T02:01:41.424 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi103.front.sepia.ceph.com 2024-04-24T02:01:41.424 DEBUG:teuthology.orchestra.run.smithi103:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-04-24T02:01:41.577 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi103.front.sepia.ceph.com 2024-04-24T02:01:41.577 DEBUG:teuthology.orchestra.run.smithi103:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-04-24T02:01:41.652 INFO:tasks.ceph:Archiving mon data... 2024-04-24T02:01:41.653 DEBUG:teuthology.misc:Transferring archived files from smithi079:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yingxin-2024-04-24_01:32:34-crimson-rados-ci-yingxin-crimson-load-obc-new-1-distro-default-smithi/7671138/data/mon.a.tgz 2024-04-24T02:01:41.653 DEBUG:teuthology.orchestra.run.smithi079:> mktemp 2024-04-24T02:01:41.668 INFO:teuthology.orchestra.run.smithi079.stdout:/tmp/tmp.WqKassMmvt 2024-04-24T02:01:41.669 DEBUG:teuthology.orchestra.run.smithi079:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.WqKassMmvt 2024-04-24T02:01:41.788 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod 0666 /tmp/tmp.WqKassMmvt 2024-04-24T02:01:41.869 DEBUG:teuthology.orchestra.remote:smithi079:/tmp/tmp.WqKassMmvt is 233KB 2024-04-24T02:01:41.964 DEBUG:teuthology.orchestra.run.smithi079:> rm -fr /tmp/tmp.WqKassMmvt 2024-04-24T02:01:41.979 INFO:tasks.ceph:Cleaning ceph cluster... 2024-04-24T02:01:41.979 DEBUG:teuthology.orchestra.run.smithi079:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-24T02:01:42.021 DEBUG:teuthology.orchestra.run.smithi103:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-24T02:01:42.096 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-24T02:01:42.111 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-24T02:01:42.111 INFO:tasks.ceph:Archiving crash dumps... 2024-04-24T02:01:42.113 DEBUG:teuthology.misc:Transferring archived files from smithi079:/var/lib/ceph/crash to /home/teuthworker/archive/yingxin-2024-04-24_01:32:34-crimson-rados-ci-yingxin-crimson-load-obc-new-1-distro-default-smithi/7671138/remote/smithi079/crash 2024-04-24T02:01:42.113 DEBUG:teuthology.orchestra.run.smithi079:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-24T02:01:42.159 DEBUG:teuthology.misc:Transferring archived files from smithi103:/var/lib/ceph/crash to /home/teuthworker/archive/yingxin-2024-04-24_01:32:34-crimson-rados-ci-yingxin-crimson-load-obc-new-1-distro-default-smithi/7671138/remote/smithi103/crash 2024-04-24T02:01:42.160 DEBUG:teuthology.orchestra.run.smithi103:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-24T02:01:42.193 INFO:tasks.ceph:Compressing logs... 2024-04-24T02:01:42.193 DEBUG:teuthology.orchestra.run.smithi079:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-24T02:01:42.204 DEBUG:teuthology.orchestra.run.smithi103:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-24T02:01:42.265 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26606.log 2024-04-24T02:01:42.266 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-04-24T02:01:42.266 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26486.log 2024-04-24T02:01:42.266 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.26606.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.26606.log.gz 2024-04-24T02:01:42.266 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-04-24T02:01:42.266 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-osd.2.log: /var/log/ceph/ceph-client.admin.26486.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-04-24T02:01:42.266 INFO:teuthology.orchestra.run.smithi103.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.26486.log.gz 2024-04-24T02:01:42.266 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26566.log 2024-04-24T02:01:42.266 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26526.log 2024-04-24T02:01:42.266 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.26566.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.26566.log.gz 2024-04-24T02:01:42.266 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.26526.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.26526.log.gz 2024-04-24T02:01:42.316 INFO:teuthology.orchestra.run.smithi103.stderr: 91.7% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-04-24T02:01:42.384 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27887.log 2024-04-24T02:01:42.385 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39886.log 2024-04-24T02:01:42.385 INFO:teuthology.orchestra.run.smithi079.stderr:gzip/var/log/ceph/ceph-client.admin.27887.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.41101.log 2024-04-24T02:01:42.385 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.27887.log.gz 2024-04-24T02:01:42.385 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.39886.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28115.log 2024-04-24T02:01:42.385 INFO:teuthology.orchestra.run.smithi079.stderr: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.39886.log.gz 2024-04-24T02:01:42.385 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.41101.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44190.log 2024-04-24T02:01:42.385 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41101.log.gz 2024-04-24T02:01:42.385 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.28115.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45473.log 2024-04-24T02:01:42.385 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28115.log.gz 2024-04-24T02:01:42.385 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.44190.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35203.log 2024-04-24T02:01:42.385 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44190.log.gz 2024-04-24T02:01:42.385 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.45473.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44230.log 2024-04-24T02:01:42.386 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.45473.log.gz 2024-04-24T02:01:42.386 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.35203.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43138.log 2024-04-24T02:01:42.386 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.35203.log.gz 2024-04-24T02:01:42.386 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.44230.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42304.log 2024-04-24T02:01:42.386 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44230.log.gz 2024-04-24T02:01:42.386 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.43138.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38075.log 2024-04-24T02:01:42.386 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43138.log.gz 2024-04-24T02:01:42.386 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.42304.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31492.log 2024-04-24T02:01:42.386 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42304.log.gz 2024-04-24T02:01:42.386 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.38075.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40064.log 2024-04-24T02:01:42.386 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.38075.log.gz 2024-04-24T02:01:42.386 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.31492.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34527.log 2024-04-24T02:01:42.386 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31492.log.gz 2024-04-24T02:01:42.386 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.40064.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37670.log 2024-04-24T02:01:42.387 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40064.log.gz 2024-04-24T02:01:42.387 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.34527.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31592.log 2024-04-24T02:01:42.387 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34527.log.gz 2024-04-24T02:01:42.387 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.37670.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31213.log 2024-04-24T02:01:42.387 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.37670.log.gz 2024-04-24T02:01:42.387 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.31592.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35968.log 2024-04-24T02:01:42.387 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31592.log.gz 2024-04-24T02:01:42.387 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39717.log 2024-04-24T02:01:42.387 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.31213.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.31213.log.gz 2024-04-24T02:01:42.387 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.35968.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39528.log 2024-04-24T02:01:42.387 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.35968.log.gz 2024-04-24T02:01:42.387 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.39717.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44070.log 2024-04-24T02:01:42.387 INFO:teuthology.orchestra.run.smithi079.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.39717.log.gz 2024-04-24T02:01:42.387 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.39528.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39359.log 2024-04-24T02:01:42.387 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.39528.log.gz 2024-04-24T02:01:42.388 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.44070.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38707.log 2024-04-24T02:01:42.388 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.44070.log.gz 2024-04-24T02:01:42.388 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.39359.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42771.log 2024-04-24T02:01:42.388 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.39359.log.gz 2024-04-24T02:01:42.388 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.38707.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40144.log 2024-04-24T02:01:42.388 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.38707.log.gz 2024-04-24T02:01:42.388 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.42771.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44950.log 2024-04-24T02:01:42.388 INFO:teuthology.orchestra.run.smithi079.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.42771.log.gz 2024-04-24T02:01:42.388 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.40144.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40691.log 2024-04-24T02:01:42.388 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40144.log.gz 2024-04-24T02:01:42.388 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.44950.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41856.log 2024-04-24T02:01:42.388 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44950.log.gz 2024-04-24T02:01:42.388 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.40691.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44992.log 2024-04-24T02:01:42.388 INFO:teuthology.orchestra.run.smithi079.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.40691.log.gz 2024-04-24T02:01:42.388 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.41856.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38516.log 2024-04-24T02:01:42.389 INFO:teuthology.orchestra.run.smithi079.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.41856.log.gz 2024-04-24T02:01:42.389 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.44992.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40711.log 2024-04-24T02:01:42.389 INFO:teuthology.orchestra.run.smithi079.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.44992.log.gz 2024-04-24T02:01:42.389 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.38516.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42087.log 2024-04-24T02:01:42.389 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.38516.log.gz 2024-04-24T02:01:42.389 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.40711.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32687.log 2024-04-24T02:01:42.389 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40711.log.gz 2024-04-24T02:01:42.389 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.42087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38774.log 2024-04-24T02:01:42.389 INFO:teuthology.orchestra.run.smithi079.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.42087.log.gz 2024-04-24T02:01:42.389 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.32687.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41647.log 2024-04-24T02:01:42.389 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32687.log.gz 2024-04-24T02:01:42.389 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.38774.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44601.log 2024-04-24T02:01:42.389 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.38774.log.gz 2024-04-24T02:01:42.389 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.41647.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36803.log 2024-04-24T02:01:42.390 INFO:teuthology.orchestra.run.smithi079.stderr: 50.8% -- replaced with /var/log/ceph/ceph-client.admin.41647.log.gz 2024-04-24T02:01:42.390 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.44601.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30519.log 2024-04-24T02:01:42.390 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.44601.log.gz 2024-04-24T02:01:42.390 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.36803.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37857.log 2024-04-24T02:01:42.390 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.36803.log.gz 2024-04-24T02:01:42.390 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.30519.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28013.log 2024-04-24T02:01:42.390 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30519.log.gz 2024-04-24T02:01:42.390 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.37857.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30906.log 2024-04-24T02:01:42.390 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.37857.log.gz 2024-04-24T02:01:42.390 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.28013.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34665.log 2024-04-24T02:01:42.390 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.28013.log.gz 2024-04-24T02:01:42.390 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.30906.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32548.log 2024-04-24T02:01:42.390 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30906.log.gz 2024-04-24T02:01:42.390 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.34665.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29231.log 2024-04-24T02:01:42.390 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34665.log.gz 2024-04-24T02:01:42.391 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.32548.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43585.log 2024-04-24T02:01:42.391 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32548.log.gz 2024-04-24T02:01:42.391 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41368.log 2024-04-24T02:01:42.391 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.29231.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29231.log.gz 2024-04-24T02:01:42.391 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.43585.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33086.log 2024-04-24T02:01:42.391 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43585.log.gz 2024-04-24T02:01:42.391 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.41368.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38158.log 2024-04-24T02:01:42.391 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41368.log.gz 2024-04-24T02:01:42.391 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.33086.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40611.log 2024-04-24T02:01:42.391 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33086.log.gz 2024-04-24T02:01:42.391 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.38158.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44488.log 2024-04-24T02:01:42.391 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.38158.log.gz 2024-04-24T02:01:42.391 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.40611.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30395.log 2024-04-24T02:01:42.391 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40611.log.gz 2024-04-24T02:01:42.392 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.44488.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45134.log 2024-04-24T02:01:42.392 INFO:teuthology.orchestra.run.smithi079.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.44488.log.gz 2024-04-24T02:01:42.392 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.30395.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32059.log 2024-04-24T02:01:42.392 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30395.log.gz 2024-04-24T02:01:42.392 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.45134.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30479.log 2024-04-24T02:01:42.392 INFO:teuthology.orchestra.run.smithi079.stderr: 50.8% -- replaced with /var/log/ceph/ceph-client.admin.45134.log.gz 2024-04-24T02:01:42.392 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29562.log 2024-04-24T02:01:42.392 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.32059.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.32059.log.gz 2024-04-24T02:01:42.392 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.30479.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33344.log 2024-04-24T02:01:42.392 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30479.log.gz 2024-04-24T02:01:42.392 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.29562.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39577.log 2024-04-24T02:01:42.392 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29562.log.gz 2024-04-24T02:01:42.392 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33304.log 2024-04-24T02:01:42.392 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.33344.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33344.log.gz 2024-04-24T02:01:42.392 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.39577.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33832.log 2024-04-24T02:01:42.393 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.39577.log.gz 2024-04-24T02:01:42.393 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.33304.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45600.log 2024-04-24T02:01:42.393 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33304.log.gz 2024-04-24T02:01:42.393 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-24T02:01:42.393 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.33832.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33832.log.gz 2024-04-24T02:01:42.393 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.45600.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27558.log 2024-04-24T02:01:42.393 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.45600.log.gz 2024-04-24T02:01:42.393 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36286.log 2024-04-24T02:01:42.393 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.27558.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41328.log 2024-04-24T02:01:42.393 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.27558.log.gz 2024-04-24T02:01:42.393 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.36286.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34805.log 2024-04-24T02:01:42.393 INFO:teuthology.orchestra.run.smithi079.stderr: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.36286.log.gz 2024-04-24T02:01:42.393 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.41328.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36097.log 2024-04-24T02:01:42.393 INFO:teuthology.orchestra.run.smithi079.stderr: 47.6% -- replaced with /var/log/ceph/ceph-client.admin.41328.log.gz 2024-04-24T02:01:42.393 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.34805.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38026.log 2024-04-24T02:01:42.394 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34805.log.gz 2024-04-24T02:01:42.394 INFO:teuthology.orchestra.run.smithi079.stderr: 86.0% -- replaced with /var/log/ceph/ceph.log.gz 2024-04-24T02:01:42.394 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.36097.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36048.log 2024-04-24T02:01:42.394 INFO:teuthology.orchestra.run.smithi079.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.36097.log.gz 2024-04-24T02:01:42.394 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.38026.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40282.log 2024-04-24T02:01:42.394 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.38026.log.gz 2024-04-24T02:01:42.394 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.36048.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35581.log 2024-04-24T02:01:42.394 INFO:teuthology.orchestra.run.smithi079.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.36048.log.gz 2024-04-24T02:01:42.394 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.40282.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34050.log 2024-04-24T02:01:42.394 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40282.log.gz 2024-04-24T02:01:42.394 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.35581.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28114.log 2024-04-24T02:01:42.394 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.35581.log.gz 2024-04-24T02:01:42.394 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35441.log/var/log/ceph/ceph-client.admin.34050.log: 2024-04-24T02:01:42.394 INFO:teuthology.orchestra.run.smithi079.stderr: 48.0% -- replaced with /var/log/ceph/ceph-client.admin.34050.log.gz 2024-04-24T02:01:42.394 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.28114.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36714.log 2024-04-24T02:01:42.395 INFO:teuthology.orchestra.run.smithi079.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.28114.log.gz 2024-04-24T02:01:42.395 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31303.log 2024-04-24T02:01:42.395 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.35441.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.35441.log.gz 2024-04-24T02:01:42.395 INFO:teuthology.orchestra.run.smithi079.stderr:gzip/var/log/ceph/ceph-client.admin.36714.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.33224.log 2024-04-24T02:01:42.395 INFO:teuthology.orchestra.run.smithi079.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.36714.log.gz 2024-04-24T02:01:42.395 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.31303.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30946.log 2024-04-24T02:01:42.395 INFO:teuthology.orchestra.run.smithi079.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.31303.log.gz 2024-04-24T02:01:42.395 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31750.log 2024-04-24T02:01:42.395 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.33224.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33224.log.gz 2024-04-24T02:01:42.395 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.30946.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43358.log 2024-04-24T02:01:42.395 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30946.log.gz 2024-04-24T02:01:42.395 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.31750.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39032.log 2024-04-24T02:01:42.395 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31750.log.gz 2024-04-24T02:01:42.395 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.43358.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32246.log 2024-04-24T02:01:42.395 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43358.log.gz 2024-04-24T02:01:42.396 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46380.log 2024-04-24T02:01:42.396 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.39032.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.39032.log.gz 2024-04-24T02:01:42.396 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.32246.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41776.log 2024-04-24T02:01:42.396 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32246.log.gz 2024-04-24T02:01:42.396 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.46380.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39399.log 2024-04-24T02:01:42.396 INFO:teuthology.orchestra.run.smithi079.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.46380.log.gz 2024-04-24T02:01:42.396 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.41776.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32778.log 2024-04-24T02:01:42.396 INFO:teuthology.orchestra.run.smithi079.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.41776.log.gz 2024-04-24T02:01:42.396 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.39399.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33616.log 2024-04-24T02:01:42.396 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.39399.log.gz 2024-04-24T02:01:42.396 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.32778.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29911.log 2024-04-24T02:01:42.396 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32778.log.gz 2024-04-24T02:01:42.396 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.33616.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43625.log 2024-04-24T02:01:42.396 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.33616.log.gz 2024-04-24T02:01:42.396 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.29911.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45303.log 2024-04-24T02:01:42.397 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29911.log.gz 2024-04-24T02:01:42.397 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.43625.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41548.log 2024-04-24T02:01:42.397 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43625.log.gz 2024-04-24T02:01:42.397 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.45303.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36367.log 2024-04-24T02:01:42.397 INFO:teuthology.orchestra.run.smithi079.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.45303.log.gz 2024-04-24T02:01:42.397 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.41548.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.29762.log 2024-04-24T02:01:42.397 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41548.log.gz 2024-04-24T02:01:42.397 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.36367.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40303.log 2024-04-24T02:01:42.397 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.36367.log.gz 2024-04-24T02:01:42.397 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.0.29762.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46170.log 2024-04-24T02:01:42.397 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.0.29762.log.gz 2024-04-24T02:01:42.397 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.40303.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34616.log 2024-04-24T02:01:42.397 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40303.log.gz 2024-04-24T02:01:42.397 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.46170.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46254.log 2024-04-24T02:01:42.398 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46170.log.gz 2024-04-24T02:01:42.398 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.34616.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32019.log 2024-04-24T02:01:42.398 INFO:teuthology.orchestra.run.smithi079.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.34616.log.gz 2024-04-24T02:01:42.398 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.46254.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45432.log 2024-04-24T02:01:42.398 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46254.log.gz 2024-04-24T02:01:42.398 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46212.log 2024-04-24T02:01:42.398 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.32019.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.32019.log.gz 2024-04-24T02:01:42.398 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.45432.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44537.log 2024-04-24T02:01:42.398 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.45432.log.gz 2024-04-24T02:01:42.398 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.46212.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-24T02:01:42.398 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46212.log.gz 2024-04-24T02:01:42.398 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.44537.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45834.log 2024-04-24T02:01:42.398 INFO:teuthology.orchestra.run.smithi079.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.44537.log.gz 2024-04-24T02:01:42.398 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28915.log 2024-04-24T02:01:42.399 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.45834.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36146.log 2024-04-24T02:01:42.399 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.45834.log.gz 2024-04-24T02:01:42.399 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.28915.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44368.log 2024-04-24T02:01:42.399 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28915.log.gz 2024-04-24T02:01:42.399 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.36146.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38912.log 2024-04-24T02:01:42.399 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.36146.log.gz 2024-04-24T02:01:42.399 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.44368.log: 88.1% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-24T02:01:42.399 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41736.log 2024-04-24T02:01:42.399 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44368.log.gz 2024-04-24T02:01:42.399 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.38912.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37937.log 2024-04-24T02:01:42.399 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.38912.log.gz 2024-04-24T02:01:42.399 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.41736.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30353.log 2024-04-24T02:01:42.399 INFO:teuthology.orchestra.run.smithi079.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.41736.log.gz 2024-04-24T02:01:42.399 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.37937.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37977.log 2024-04-24T02:01:42.399 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.37937.log.gz 2024-04-24T02:01:42.400 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.30353.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43009.log 2024-04-24T02:01:42.400 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30353.log.gz 2024-04-24T02:01:42.400 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.37977.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37719.log 2024-04-24T02:01:42.400 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.37977.log.gz 2024-04-24T02:01:42.400 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.43009.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39439.log 2024-04-24T02:01:42.400 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43009.log.gz 2024-04-24T02:01:42.400 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.37719.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45748.log 2024-04-24T02:01:42.400 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.37719.log.gz 2024-04-24T02:01:42.400 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.39439.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41408.log 2024-04-24T02:01:42.400 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.39439.log.gz 2024-04-24T02:01:42.400 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.45748.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41506.log 2024-04-24T02:01:42.400 INFO:teuthology.orchestra.run.smithi079.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.45748.log.gz 2024-04-24T02:01:42.400 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.41408.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36674.log 2024-04-24T02:01:42.400 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41408.log.gz 2024-04-24T02:01:42.401 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.41506.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30995.log 2024-04-24T02:01:42.401 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41506.log.gz 2024-04-24T02:01:42.401 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33666.log 2024-04-24T02:01:42.401 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.36674.log: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.36674.log.gz 2024-04-24T02:01:42.401 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.30995.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40471.log 2024-04-24T02:01:42.401 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30995.log.gz 2024-04-24T02:01:42.401 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.33666.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31799.log 2024-04-24T02:01:42.401 INFO:teuthology.orchestra.run.smithi079.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.33666.log.gz 2024-04-24T02:01:42.401 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.40471.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42731.log 2024-04-24T02:01:42.401 INFO:teuthology.orchestra.run.smithi079.stderr: 48.0% -- replaced with /var/log/ceph/ceph-client.admin.40471.log.gz 2024-04-24T02:01:42.401 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.31799.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33750.log 2024-04-24T02:01:42.401 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31799.log.gz 2024-04-24T02:01:42.401 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.42731.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43456.log 2024-04-24T02:01:42.401 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42731.log.gz 2024-04-24T02:01:42.401 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.33750.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46086.log 2024-04-24T02:01:42.402 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.33750.log.gz 2024-04-24T02:01:42.402 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.43456.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46128.log 2024-04-24T02:01:42.402 INFO:teuthology.orchestra.run.smithi079.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.43456.log.gz 2024-04-24T02:01:42.402 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.46086.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32827.log 2024-04-24T02:01:42.402 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46086.log.gz 2024-04-24T02:01:42.402 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.46128.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44021.log 2024-04-24T02:01:42.402 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46128.log.gz 2024-04-24T02:01:42.402 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.32827.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33046.log 2024-04-24T02:01:42.402 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32827.log.gz 2024-04-24T02:01:42.402 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.44021.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32350.log 2024-04-24T02:01:42.402 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44021.log.gz 2024-04-24T02:01:42.402 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.33046.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34418.log 2024-04-24T02:01:42.402 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33046.log.gz 2024-04-24T02:01:42.403 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.32350.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27837.log 2024-04-24T02:01:42.403 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.32350.log.gz 2024-04-24T02:01:42.403 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.34418.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28283.log 2024-04-24T02:01:42.403 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.34418.log.gz 2024-04-24T02:01:42.403 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.27837.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28895.log 2024-04-24T02:01:42.403 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.27837.log.gz 2024-04-24T02:01:42.403 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.28283.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33264.log 2024-04-24T02:01:42.403 INFO:teuthology.orchestra.run.smithi079.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.28283.log.gz 2024-04-24T02:01:42.403 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.28895.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28559.log 2024-04-24T02:01:42.403 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.28895.log.gz 2024-04-24T02:01:42.403 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.33264.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42169.log 2024-04-24T02:01:42.403 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.33264.log.gz 2024-04-24T02:01:42.403 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.28559.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37159.log 2024-04-24T02:01:42.403 INFO:teuthology.orchestra.run.smithi079.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.28559.log.gz 2024-04-24T02:01:42.403 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.42169.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43803.log 2024-04-24T02:01:42.404 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42169.log.gz 2024-04-24T02:01:42.404 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.37159.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33892.log 2024-04-24T02:01:42.404 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.37159.log.gz 2024-04-24T02:01:42.404 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.43803.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42344.log 2024-04-24T02:01:42.404 INFO:teuthology.orchestra.run.smithi079.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.43803.log.gz 2024-04-24T02:01:42.404 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.33892.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41150.log 2024-04-24T02:01:42.404 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33892.log.gz 2024-04-24T02:01:42.404 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.42344.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29732.log 2024-04-24T02:01:42.404 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42344.log.gz 2024-04-24T02:01:42.404 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.41150.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37390.log 2024-04-24T02:01:42.404 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41150.log.gz 2024-04-24T02:01:42.404 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.29732.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31084.log 2024-04-24T02:01:42.404 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29732.log.gz 2024-04-24T02:01:42.404 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.37390.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35294.log 2024-04-24T02:01:42.405 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.37390.log.gz 2024-04-24T02:01:42.405 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.31084.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35532.log 2024-04-24T02:01:42.405 INFO:teuthology.orchestra.run.smithi079.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.31084.log.gz 2024-04-24T02:01:42.405 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.35294.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28431.log 2024-04-24T02:01:42.405 INFO:teuthology.orchestra.run.smithi079.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.35294.log.gz 2024-04-24T02:01:42.405 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.35532.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28282.log 2024-04-24T02:01:42.405 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.35532.log.gz 2024-04-24T02:01:42.405 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.28431.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29063.log 2024-04-24T02:01:42.405 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.28431.log.gz 2024-04-24T02:01:42.405 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.28282.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27529.log 2024-04-24T02:01:42.405 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.28282.log.gz 2024-04-24T02:01:42.405 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.29063.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31352.log 2024-04-24T02:01:42.405 INFO:teuthology.orchestra.run.smithi079.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.29063.log.gz 2024-04-24T02:01:42.405 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.27529.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30262.log 2024-04-24T02:01:42.405 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.27529.log.gz 2024-04-24T02:01:42.406 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.31352.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32897.log 2024-04-24T02:01:42.406 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31352.log.gz 2024-04-24T02:01:42.406 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.30262.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36008.log 2024-04-24T02:01:42.406 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30262.log.gz 2024-04-24T02:01:42.406 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.32897.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37021.log 2024-04-24T02:01:42.406 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.32897.log.gz 2024-04-24T02:01:42.406 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.36008.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30866.log 2024-04-24T02:01:42.406 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.36008.log.gz 2024-04-24T02:01:42.406 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.37021.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28430.log 2024-04-24T02:01:42.406 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.37021.log.gz 2024-04-24T02:01:42.406 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.30866.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36495.log 2024-04-24T02:01:42.406 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30866.log.gz 2024-04-24T02:01:42.406 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.28430.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32736.log 2024-04-24T02:01:42.406 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28430.log.gz 2024-04-24T02:01:42.407 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.36495.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46462.log 2024-04-24T02:01:42.407 INFO:teuthology.orchestra.run.smithi079.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.36495.log.gz 2024-04-24T02:01:42.407 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.32736.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28281.log 2024-04-24T02:01:42.407 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.32736.log.gz 2024-04-24T02:01:42.407 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.46462.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-04-24T02:01:42.407 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46462.log.gz 2024-04-24T02:01:42.407 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.28281.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43098.log 2024-04-24T02:01:42.407 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28281.log.gz 2024-04-24T02:01:42.407 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35483.log 2024-04-24T02:01:42.407 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-osd.1.log: /var/log/ceph/ceph-client.admin.43098.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28787.log 2024-04-24T02:01:42.407 INFO:teuthology.orchestra.run.smithi079.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.43098.log.gz 2024-04-24T02:01:42.407 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.35483.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37341.log 2024-04-24T02:01:42.407 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.35483.log.gz 2024-04-24T02:01:42.407 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.28787.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37061.log 2024-04-24T02:01:42.408 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28787.log.gz 2024-04-24T02:01:42.408 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.37341.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33792.log 2024-04-24T02:01:42.408 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.37341.log.gz 2024-04-24T02:01:42.408 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.37061.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32962.log 2024-04-24T02:01:42.408 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.37061.log.gz 2024-04-24T02:01:42.408 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.33792.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-04-24T02:01:42.408 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.33792.log.gz 2024-04-24T02:01:42.408 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.32962.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37628.log 2024-04-24T02:01:42.408 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32962.log.gz 2024-04-24T02:01:42.408 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39975.log 2024-04-24T02:01:42.408 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.37628.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30303.log 2024-04-24T02:01:42.408 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.37628.log.gz 2024-04-24T02:01:42.408 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.39975.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44279.log 2024-04-24T02:01:42.408 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.39975.log.gz 2024-04-24T02:01:42.408 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.30303.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41059.log 2024-04-24T02:01:42.409 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30303.log.gz 2024-04-24T02:01:42.409 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.44279.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40422.log 2024-04-24T02:01:42.409 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44279.log.gz 2024-04-24T02:01:42.409 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.41059.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40919.log 2024-04-24T02:01:42.409 INFO:teuthology.orchestra.run.smithi079.stderr: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.41059.log.gz 2024-04-24T02:01:42.409 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.40422.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36754.log 2024-04-24T02:01:42.409 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40422.log.gz 2024-04-24T02:01:42.409 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.40919.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32596.log 2024-04-24T02:01:42.409 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40919.log.gz 2024-04-24T02:01:42.409 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.36754.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36594.log 2024-04-24T02:01:42.409 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.36754.log.gz 2024-04-24T02:01:42.409 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.32596.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32596.log.gz 2024-04-24T02:01:42.409 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39270.log 2024-04-24T02:01:42.409 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.36594.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39130.log 2024-04-24T02:01:42.409 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.36594.log.gz 2024-04-24T02:01:42.410 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.39270.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35928.log 2024-04-24T02:01:42.410 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.39270.log.gz 2024-04-24T02:01:42.410 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44408.log 2024-04-24T02:01:42.410 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.39130.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.39130.log.gz 2024-04-24T02:01:42.410 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.35928.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37201.log 2024-04-24T02:01:42.410 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.35928.log.gz 2024-04-24T02:01:42.410 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.44408.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46296.log 2024-04-24T02:01:42.410 INFO:teuthology.orchestra.run.smithi079.stderr: 50.8% -- replaced with /var/log/ceph/ceph-client.admin.44408.log.gz 2024-04-24T02:01:42.410 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.37201.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30737.log 2024-04-24T02:01:42.410 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.37201.log.gz 2024-04-24T02:01:42.410 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.46296.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39172.log 2024-04-24T02:01:42.410 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46296.log.gz 2024-04-24T02:01:42.410 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.30737.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35392.log 2024-04-24T02:01:42.410 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30737.log.gz 2024-04-24T02:01:42.411 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.39172.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40821.log 2024-04-24T02:01:42.411 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.39172.log.gz 2024-04-24T02:01:42.411 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.35392.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33184.log 2024-04-24T02:01:42.411 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.35392.log.gz 2024-04-24T02:01:42.411 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.40821.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34925.log 2024-04-24T02:01:42.411 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40821.log.gz 2024-04-24T02:01:42.411 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.33184.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42513.log 2024-04-24T02:01:42.411 INFO:teuthology.orchestra.run.smithi079.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.33184.log.gz 2024-04-24T02:01:42.411 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.34925.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28727.log 2024-04-24T02:01:42.411 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34925.log.gz 2024-04-24T02:01:42.411 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.42513.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40184.log 2024-04-24T02:01:42.411 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42513.log.gz 2024-04-24T02:01:42.411 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.28727.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30129.log 2024-04-24T02:01:42.411 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28727.log.gz 2024-04-24T02:01:42.412 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.40184.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40373.log 2024-04-24T02:01:42.412 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40184.log.gz 2024-04-24T02:01:42.412 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.30129.log: gzip 50.0% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.30129.log.gz 2024-04-24T02:01:42.412 INFO:teuthology.orchestra.run.smithi079.stderr: -- /var/log/ceph/ceph-client.admin.38291.log 2024-04-24T02:01:42.412 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.40373.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34179.log 2024-04-24T02:01:42.412 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40373.log.gz 2024-04-24T02:01:42.412 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.38291.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34756.log 2024-04-24T02:01:42.412 INFO:teuthology.orchestra.run.smithi079.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.38291.log.gz 2024-04-24T02:01:42.412 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.34179.log: 50.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.34179.log.gz 2024-04-24T02:01:42.412 INFO:teuthology.orchestra.run.smithi079.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.27781.log 2024-04-24T02:01:42.412 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.34756.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40562.log 2024-04-24T02:01:42.412 INFO:teuthology.orchestra.run.smithi079.stderr: 48.0% -- replaced with /var/log/ceph/ceph-client.admin.34756.log.gz 2024-04-24T02:01:42.412 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.27781.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34398.log 2024-04-24T02:01:42.413 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.27781.log.gz 2024-04-24T02:01:42.413 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.40562.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41199.log 2024-04-24T02:01:42.413 INFO:teuthology.orchestra.run.smithi079.stderr: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.40562.log.gz 2024-04-24T02:01:42.413 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.34398.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34309.log 2024-04-24T02:01:42.413 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34398.log.gz 2024-04-24T02:01:42.413 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.41199.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36913.log 2024-04-24T02:01:42.413 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41199.log.gz 2024-04-24T02:01:42.413 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.34309.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45791.log 2024-04-24T02:01:42.413 INFO:teuthology.orchestra.run.smithi079.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.34309.log.gz 2024-04-24T02:01:42.413 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.36913.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44770.log 2024-04-24T02:01:42.413 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.36913.log.gz 2024-04-24T02:01:42.413 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.45791.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27373.log 2024-04-24T02:01:42.413 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.45791.log.gz 2024-04-24T02:01:42.413 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.44770.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45263.log 2024-04-24T02:01:42.413 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44770.log.gz 2024-04-24T02:01:42.414 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.27373.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42691.log 2024-04-24T02:01:42.414 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.27373.log.gz 2024-04-24T02:01:42.414 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.45263.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35014.log 2024-04-24T02:01:42.414 INFO:teuthology.orchestra.run.smithi079.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.45263.log.gz 2024-04-24T02:01:42.414 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.42691.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31044.log 2024-04-24T02:01:42.414 INFO:teuthology.orchestra.run.smithi079.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.42691.log.gz 2024-04-24T02:01:42.414 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.35014.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42960.log 2024-04-24T02:01:42.414 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.35014.log.gz 2024-04-24T02:01:42.414 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.31044.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41457.log 2024-04-24T02:01:42.414 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31044.log.gz 2024-04-24T02:01:42.414 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33442.log 2024-04-24T02:01:42.414 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.42960.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42960.log.gz 2024-04-24T02:01:42.414 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.41457.log: gzip -5 --verbose -- 50.0% /var/log/ceph/ceph-client.admin.41905.log -- replaced with /var/log/ceph/ceph-client.admin.41457.log.gz 2024-04-24T02:01:42.414 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-24T02:01:42.415 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.33442.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38872.log 2024-04-24T02:01:42.415 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33442.log.gz 2024-04-24T02:01:42.415 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.41905.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40024.log 2024-04-24T02:01:42.415 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41905.log.gz 2024-04-24T02:01:42.415 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.38872.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36872.log 2024-04-24T02:01:42.415 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.38872.log.gz 2024-04-24T02:01:42.415 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.40024.log: gzip -5 --verbose -- 50.0% /var/log/ceph/ceph-client.admin.28112.log -- replaced with /var/log/ceph/ceph-client.admin.40024.log.gz 2024-04-24T02:01:42.415 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-24T02:01:42.415 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.36872.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32288.log 2024-04-24T02:01:42.415 INFO:teuthology.orchestra.run.smithi079.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.36872.log.gz 2024-04-24T02:01:42.415 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.28112.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45136.log 2024-04-24T02:01:42.415 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28112.log.gz 2024-04-24T02:01:42.415 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.32288.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33006.log 2024-04-24T02:01:42.415 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32288.log.gz 2024-04-24T02:01:42.415 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.45136.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42611.log 2024-04-24T02:01:42.416 INFO:teuthology.orchestra.run.smithi079.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.45136.log.gz 2024-04-24T02:01:42.416 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.33006.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43665.log 2024-04-24T02:01:42.416 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33006.log.gz 2024-04-24T02:01:42.416 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.42611.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45622.log 2024-04-24T02:01:42.416 INFO:teuthology.orchestra.run.smithi079.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.42611.log.gz 2024-04-24T02:01:42.416 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.43665.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44686.log 2024-04-24T02:01:42.416 INFO:teuthology.orchestra.run.smithi079.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.43665.log.gz 2024-04-24T02:01:42.416 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.45622.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-04-24T02:01:42.416 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.45622.log.gz 2024-04-24T02:01:42.416 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.44686.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35621.log 2024-04-24T02:01:42.416 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44686.log.gz 2024-04-24T02:01:42.416 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40870.log 2024-04-24T02:01:42.416 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.35621.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34099.log 2024-04-24T02:01:42.416 INFO:teuthology.orchestra.run.smithi079.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.35621.log.gz 2024-04-24T02:01:42.417 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.40870.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43545.log 2024-04-24T02:01:42.417 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40870.log.gz 2024-04-24T02:01:42.417 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29815.log 2024-04-24T02:01:42.417 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.34099.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.34099.log.gz 2024-04-24T02:01:42.417 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.43545.log: gzip 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43545.log.gz 2024-04-24T02:01:42.417 INFO:teuthology.orchestra.run.smithi079.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.36634.log 2024-04-24T02:01:42.417 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.29815.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45918.log 2024-04-24T02:01:42.417 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29815.log.gz 2024-04-24T02:01:42.417 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.36634.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46338.log 2024-04-24T02:01:42.417 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.36634.log.gz 2024-04-24T02:01:42.417 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.45918.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46002.log 2024-04-24T02:01:42.417 INFO:teuthology.orchestra.run.smithi079.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.45918.log.gz 2024-04-24T02:01:42.417 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.46338.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43058.log 2024-04-24T02:01:42.418 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46338.log.gz 2024-04-24T02:01:42.418 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.46002.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46044.log 2024-04-24T02:01:42.418 INFO:teuthology.orchestra.run.smithi079.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.46002.log.gz 2024-04-24T02:01:42.418 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.43058.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35154.log 2024-04-24T02:01:42.418 INFO:teuthology.orchestra.run.smithi079.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.43058.log.gz 2024-04-24T02:01:42.418 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.46044.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31979.log 2024-04-24T02:01:42.418 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46044.log.gz 2024-04-24T02:01:42.418 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.35154.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45960.log 2024-04-24T02:01:42.418 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.35154.log.gz 2024-04-24T02:01:42.418 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.31979.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35063.log 2024-04-24T02:01:42.418 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31979.log.gz 2024-04-24T02:01:42.418 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.45960.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38565.log 2024-04-24T02:01:42.418 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.45960.log.gz 2024-04-24T02:01:42.418 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.35063.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34480.log 2024-04-24T02:01:42.419 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.35063.log.gz 2024-04-24T02:01:42.419 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36237.log 2024-04-24T02:01:42.419 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.38565.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.38565.log.gz 2024-04-24T02:01:42.419 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.34480.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29273.log 2024-04-24T02:01:42.419 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.34480.log.gz 2024-04-24T02:01:42.419 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36545.log 2024-04-24T02:01:42.419 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.36237.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.36237.log.gz 2024-04-24T02:01:42.419 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.29273.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30080.log 2024-04-24T02:01:42.419 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29273.log.gz 2024-04-24T02:01:42.419 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.36545.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45412.log 2024-04-24T02:01:42.419 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.36545.log.gz 2024-04-24T02:01:42.419 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33961.log 2024-04-24T02:01:42.419 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.30080.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30080.log.gz 2024-04-24T02:01:42.419 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.45412.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29315.log 2024-04-24T02:01:42.420 INFO:teuthology.orchestra.run.smithi079.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.45412.log.gz 2024-04-24T02:01:42.420 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.33961.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37110.log 2024-04-24T02:01:42.420 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33961.log.gz 2024-04-24T02:01:42.420 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33462.log 2024-04-24T02:01:42.420 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.29315.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29315.log.gz 2024-04-24T02:01:42.420 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.37110.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29474.log 2024-04-24T02:01:42.420 INFO:teuthology.orchestra.run.smithi079.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.37110.log.gz 2024-04-24T02:01:42.420 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.33462.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31104.log 2024-04-24T02:01:42.420 INFO:teuthology.orchestra.run.smithi079.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.33462.log.gz 2024-04-24T02:01:42.420 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38250.log 2024-04-24T02:01:42.420 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.29474.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29474.log.gz 2024-04-24T02:01:42.420 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.31104.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35859.log 2024-04-24T02:01:42.420 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31104.log.gz 2024-04-24T02:01:42.420 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42651.log 2024-04-24T02:01:42.420 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.38250.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.38250.log.gz 2024-04-24T02:01:42.421 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.35859.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28598.log 2024-04-24T02:01:42.421 INFO:teuthology.orchestra.run.smithi079.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.35859.log.gz 2024-04-24T02:01:42.421 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.42651.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28391.log 2024-04-24T02:01:42.421 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42651.log.gz 2024-04-24T02:01:42.421 INFO:teuthology.orchestra.run.smithi079.stderr:gzip/var/log/ceph/ceph-client.admin.28598.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.40233.log 2024-04-24T02:01:42.421 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.28598.log.gz 2024-04-24T02:01:42.421 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.28391.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44328.log 2024-04-24T02:01:42.421 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28391.log.gz 2024-04-24T02:01:42.421 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30539.log 2024-04-24T02:01:42.421 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.40233.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40233.log.gz 2024-04-24T02:01:42.421 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.44328.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30437.log 2024-04-24T02:01:42.421 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44328.log.gz 2024-04-24T02:01:42.421 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40961.log 2024-04-24T02:01:42.421 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.30539.log: 48.4% -- replaced with /var/log/ceph/ceph-client.admin.30539.log.gz 2024-04-24T02:01:42.421 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.30437.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34945.log 2024-04-24T02:01:42.422 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30437.log.gz 2024-04-24T02:01:42.422 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.40961.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41995.log 2024-04-24T02:01:42.422 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40961.log.gz 2024-04-24T02:01:42.422 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38383.log 2024-04-24T02:01:42.422 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.34945.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34945.log.gz 2024-04-24T02:01:42.422 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.41995.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37439.log 2024-04-24T02:01:42.422 INFO:teuthology.orchestra.run.smithi079.stderr: 50.8% -- replaced with /var/log/ceph/ceph-client.admin.41995.log.gz 2024-04-24T02:01:42.422 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42911.log 2024-04-24T02:01:42.422 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.38383.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.38383.log.gz 2024-04-24T02:01:42.422 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.37439.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33575.log 2024-04-24T02:01:42.422 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.37439.log.gz 2024-04-24T02:01:42.422 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.42911.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44728.log 2024-04-24T02:01:42.422 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42911.log.gz 2024-04-24T02:01:42.422 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34259.log 2024-04-24T02:01:42.423 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.33575.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33575.log.gz 2024-04-24T02:01:42.423 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.44728.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29951.log 2024-04-24T02:01:42.423 INFO:teuthology.orchestra.run.smithi079.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.44728.log.gz 2024-04-24T02:01:42.423 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.34259.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39794.log 2024-04-24T02:01:42.423 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34259.log.gz 2024-04-24T02:01:42.423 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.29951.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29476.log 2024-04-24T02:01:42.423 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29951.log.gz 2024-04-24T02:01:42.423 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.39794.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31939.log 50.0% 2024-04-24T02:01:42.423 INFO:teuthology.orchestra.run.smithi079.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39794.log.gz 2024-04-24T02:01:42.423 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38117.log 2024-04-24T02:01:42.423 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.29476.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29476.log.gz 2024-04-24T02:01:42.423 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.31939.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32638.log 2024-04-24T02:01:42.423 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31939.log.gz 2024-04-24T02:01:42.423 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.38117.log: gzip -5 --verbose 49.6% -- replaced with /var/log/ceph/ceph-client.admin.38117.log.gz 2024-04-24T02:01:42.423 INFO:teuthology.orchestra.run.smithi079.stderr: -- /var/log/ceph/ceph-client.admin.27294.log 2024-04-24T02:01:42.424 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.32638.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37530.log 2024-04-24T02:01:42.424 INFO:teuthology.orchestra.run.smithi079.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.32638.log.gz 2024-04-24T02:01:42.424 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.27294.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42129.log 2024-04-24T02:01:42.424 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.27294.log.gz 2024-04-24T02:01:42.424 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.37530.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29690.log 2024-04-24T02:01:42.424 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.37530.log.gz 2024-04-24T02:01:42.424 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.42129.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45133.log 2024-04-24T02:01:42.424 INFO:teuthology.orchestra.run.smithi079.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.42129.log.gz 2024-04-24T02:01:42.424 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.29690.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31841.log 2024-04-24T02:01:42.424 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29690.log.gz 2024-04-24T02:01:42.424 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.45133.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44854.log 2024-04-24T02:01:42.424 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.45133.log.gz 2024-04-24T02:01:42.424 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.31841.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31532.log 2024-04-24T02:01:42.424 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31841.log.gz 2024-04-24T02:01:42.424 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.44854.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32349.log 2024-04-24T02:01:42.425 INFO:teuthology.orchestra.run.smithi079.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.44854.log.gz 2024-04-24T02:01:42.425 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43316.log 2024-04-24T02:01:42.425 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.31532.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.31532.log.gz 2024-04-24T02:01:42.425 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.32349.log: gzip -5 49.6% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.32349.log.gz /var/log/ceph/ceph-client.admin.33852.log 2024-04-24T02:01:42.425 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-24T02:01:42.425 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43267.log 2024-04-24T02:01:42.425 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.43316.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.43316.log.gz 2024-04-24T02:01:42.425 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.33852.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42820.log 2024-04-24T02:01:42.425 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33852.log.gz 2024-04-24T02:01:42.425 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.43267.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46422.log 2024-04-24T02:01:42.425 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43267.log.gz 2024-04-24T02:01:42.425 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38467.log 2024-04-24T02:01:42.425 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.42820.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42820.log.gz 2024-04-24T02:01:42.425 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.46422.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35661.log 2024-04-24T02:01:42.426 INFO:teuthology.orchestra.run.smithi079.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.46422.log.gz 2024-04-24T02:01:42.426 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.38467.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44812.log 2024-04-24T02:01:42.426 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.38467.log.gz 2024-04-24T02:01:42.426 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30826.log 2024-04-24T02:01:42.426 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.35661.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.35661.log.gz 2024-04-24T02:01:42.426 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.44812.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42424.log 2024-04-24T02:01:42.426 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44812.log.gz 2024-04-24T02:01:42.426 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.30826.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38685.log 2024-04-24T02:01:42.426 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30826.log.gz 2024-04-24T02:01:42.426 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40104.log 2024-04-24T02:01:42.426 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.42424.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.42424.log.gz 2024-04-24T02:01:42.426 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.38685.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29377.log 2024-04-24T02:01:42.426 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.38685.log.gz 2024-04-24T02:01:42.426 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.40104.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34139.log 2024-04-24T02:01:42.426 INFO:teuthology.orchestra.run.smithi079.stderr: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.40104.log.gz 2024-04-24T02:01:42.427 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.29377.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38208.log 2024-04-24T02:01:42.427 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29377.log.gz 2024-04-24T02:01:42.427 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.34139.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31443.log 2024-04-24T02:01:42.427 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34139.log.gz 2024-04-24T02:01:42.427 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37488.log 2024-04-24T02:01:42.427 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.38208.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.38208.log.gz 2024-04-24T02:01:42.427 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.31443.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36415.log 2024-04-24T02:01:42.427 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31443.log.gz 2024-04-24T02:01:42.427 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.37488.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45474.log 2024-04-24T02:01:42.427 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.37488.log.gz 2024-04-24T02:01:42.427 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.36415.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43178.log 2024-04-24T02:01:42.427 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.36415.log.gz 2024-04-24T02:01:42.427 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.45474.log: gzip 49.6% -- replaced with /var/log/ceph/ceph-client.admin.45474.log.gz 2024-04-24T02:01:42.427 INFO:teuthology.orchestra.run.smithi079.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.34358.log 2024-04-24T02:01:42.427 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.43178.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30786.log 2024-04-24T02:01:42.428 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.43178.log.gz 2024-04-24T02:01:42.428 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.34358.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33135.log 2024-04-24T02:01:42.428 INFO:teuthology.orchestra.run.smithi079.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.34358.log.gz 2024-04-24T02:01:42.428 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.30786.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33533.log 2024-04-24T02:01:42.428 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30786.log.gz 2024-04-24T02:01:42.428 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.33135.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45876.log 2024-04-24T02:01:42.428 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.33135.log.gz 2024-04-24T02:01:42.428 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.33533.log: 49.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.33533.log.gz 2024-04-24T02:01:42.428 INFO:teuthology.orchestra.run.smithi079.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.40780.log 2024-04-24T02:01:42.428 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44644.log 2024-04-24T02:01:42.428 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.45876.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.45876.log.gz 2024-04-24T02:01:42.428 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.40780.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32148.log 2024-04-24T02:01:42.428 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40780.log.gz 2024-04-24T02:01:42.428 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30170.log 2024-04-24T02:01:42.428 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.44644.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44644.log.gz 2024-04-24T02:01:42.429 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.32148.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35839.log 2024-04-24T02:01:42.429 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32148.log.gz 2024-04-24T02:01:42.429 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.30170.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35343.log 2024-04-24T02:01:42.429 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30170.log.gz 2024-04-24T02:01:42.429 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28599.log 2024-04-24T02:01:42.429 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.35839.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.35839.log.gz 2024-04-24T02:01:42.429 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.35343.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45304.log 2024-04-24T02:01:42.429 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.35343.log.gz 2024-04-24T02:01:42.429 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.28599.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34885.log 2024-04-24T02:01:42.429 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28599.log.gz 2024-04-24T02:01:42.429 INFO:teuthology.orchestra.run.smithi079.stderr:gzip/var/log/ceph/ceph-client.admin.45304.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.29147.log 2024-04-24T02:01:42.429 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.45304.log.gz 2024-04-24T02:01:42.429 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.34885.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42562.log 2024-04-24T02:01:42.429 INFO:teuthology.orchestra.run.smithi079.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.34885.log.gz 2024-04-24T02:01:42.429 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36188.log 2024-04-24T02:01:42.430 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.29147.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29147.log.gz 2024-04-24T02:01:42.430 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.42562.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31254.log 2024-04-24T02:01:42.430 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42562.log.gz 2024-04-24T02:01:42.430 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.36188.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39926.log 2024-04-24T02:01:42.430 INFO:teuthology.orchestra.run.smithi079.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.36188.log.gz 2024-04-24T02:01:42.430 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41816.log 2024-04-24T02:01:42.430 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.31254.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31254.log.gz 2024-04-24T02:01:42.430 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.39926.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39619.log 2024-04-24T02:01:42.430 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.39926.log.gz 2024-04-24T02:01:42.430 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.41816.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43407.log 2024-04-24T02:01:42.430 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41816.log.gz 2024-04-24T02:01:42.430 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42262.log 2024-04-24T02:01:42.430 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.39619.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.39619.log.gz 2024-04-24T02:01:42.430 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.43407.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44110.log 2024-04-24T02:01:42.431 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43407.log.gz 2024-04-24T02:01:42.431 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.42262.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39221.log 2024-04-24T02:01:42.431 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42262.log.gz 2024-04-24T02:01:42.431 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39737.log 2024-04-24T02:01:42.431 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.44110.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44110.log.gz 2024-04-24T02:01:42.431 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.39221.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35701.log 2024-04-24T02:01:42.431 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.39221.log.gz 2024-04-24T02:01:42.431 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.39737.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42464.log 2024-04-24T02:01:42.431 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.39737.log.gz 2024-04-24T02:01:42.431 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.35701.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44448.log 2024-04-24T02:01:42.431 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.35701.log.gz 2024-04-24T02:01:42.431 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.42464.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.42464.log.gz 2024-04-24T02:01:42.431 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35790.log 2024-04-24T02:01:42.431 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39668.log 2024-04-24T02:01:42.431 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.44448.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44448.log.gz 2024-04-24T02:01:42.432 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.35790.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28956.log 2024-04-24T02:01:42.432 INFO:teuthology.orchestra.run.smithi079.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.35790.log.gz 2024-04-24T02:01:42.432 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.39668.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35112.log 2024-04-24T02:01:42.432 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.39668.log.gz 2024-04-24T02:01:42.432 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31659.log 2024-04-24T02:01:42.432 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.28956.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.28956.log.gz 2024-04-24T02:01:42.432 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.35112.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45660.log 2024-04-24T02:01:42.432 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.35112.log.gz 2024-04-24T02:01:42.432 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.31659.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37788.log 2024-04-24T02:01:42.432 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31659.log.gz 2024-04-24T02:01:42.432 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43218.log 2024-04-24T02:01:42.432 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.45660.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.45660.log.gz 2024-04-24T02:01:42.432 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.37788.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38823.log 2024-04-24T02:01:42.432 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.37788.log.gz 2024-04-24T02:01:42.433 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.43218.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42220.log 2024-04-24T02:01:42.433 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43218.log.gz 2024-04-24T02:01:42.433 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.38823.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33708.log 2024-04-24T02:01:42.433 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.38823.log.gz 2024-04-24T02:01:42.433 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.42220.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32079.log 2024-04-24T02:01:42.433 INFO:teuthology.orchestra.run.smithi079.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.42220.log.gz 2024-04-24T02:01:42.433 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39319.log 2024-04-24T02:01:42.433 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.33708.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.33708.log.gz 2024-04-24T02:01:42.433 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.32079.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42384.log 2024-04-24T02:01:42.433 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.32079.log.gz 2024-04-24T02:01:42.433 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.39319.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29871.log 2024-04-24T02:01:42.433 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.39319.log.gz 2024-04-24T02:01:42.433 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28785.log 2024-04-24T02:01:42.433 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.42384.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42384.log.gz 2024-04-24T02:01:42.434 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.29871.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40513.log 2024-04-24T02:01:42.434 INFO:teuthology.orchestra.run.smithi079.stderr: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.29871.log.gz 2024-04-24T02:01:42.434 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.28785.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28113.log 2024-04-24T02:01:42.434 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28785.log.gz 2024-04-24T02:01:42.434 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.40513.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32197.log 2024-04-24T02:01:42.434 INFO:teuthology.orchestra.run.smithi079.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.40513.log.gz 2024-04-24T02:01:42.435 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.28113.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39846.log 2024-04-24T02:01:42.435 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28113.log.gz 2024-04-24T02:01:42.435 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.32197.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31566.log 2024-04-24T02:01:42.435 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32197.log.gz 2024-04-24T02:01:42.435 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.39846.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36852.log 2024-04-24T02:01:42.435 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.39846.log.gz 2024-04-24T02:01:42.435 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.31566.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45264.log 2024-04-24T02:01:42.435 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31566.log.gz 2024-04-24T02:01:42.435 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.36852.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40651.log 2024-04-24T02:01:42.435 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.36852.log.gz 2024-04-24T02:01:42.435 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.45264.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37768.log 2024-04-24T02:01:42.435 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.45264.log.gz 2024-04-24T02:01:42.435 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35252.log 2024-04-24T02:01:42.436 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.40651.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40651.log.gz 2024-04-24T02:01:42.436 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35741.log 2024-04-24T02:01:42.436 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.37768.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.37768.log.gz 2024-04-24T02:01:42.436 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.35252.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32896.log 2024-04-24T02:01:42.436 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.35252.log.gz 2024-04-24T02:01:42.436 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33393.log 2024-04-24T02:01:42.436 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.35741.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.35741.log.gz 2024-04-24T02:01:42.436 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.32896.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45034.log 2024-04-24T02:01:42.436 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32896.log.gz 2024-04-24T02:01:42.436 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.33393.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34001.log 2024-04-24T02:01:42.436 INFO:teuthology.orchestra.run.smithi079.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.33393.log.gz 2024-04-24T02:01:42.436 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.45034.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37579.log 2024-04-24T02:01:42.436 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.45034.log.gz 2024-04-24T02:01:42.436 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.34001.log: 50.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.34001.log.gz 2024-04-24T02:01:42.437 INFO:teuthology.orchestra.run.smithi079.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.43972.log 2024-04-24T02:01:42.437 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29608.log 2024-04-24T02:01:42.437 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.37579.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.37579.log.gz 2024-04-24T02:01:42.437 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.43972.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42869.log 2024-04-24T02:01:42.437 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43972.log.gz 2024-04-24T02:01:42.437 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.29608.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38645.log 2024-04-24T02:01:42.437 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29608.log.gz 2024-04-24T02:01:42.437 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39479.log 2024-04-24T02:01:42.437 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.42869.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42869.log.gz 2024-04-24T02:01:42.437 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.38645.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30031.log 2024-04-24T02:01:42.437 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.38645.log.gz 2024-04-24T02:01:42.437 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.39479.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41288.log 2024-04-24T02:01:42.437 INFO:teuthology.orchestra.run.smithi079.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.39479.log.gz 2024-04-24T02:01:42.437 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.30031.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43714.log 2024-04-24T02:01:42.438 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30031.log.gz 2024-04-24T02:01:42.438 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.41288.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28223.log 2024-04-24T02:01:42.438 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41288.log.gz 2024-04-24T02:01:42.438 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.43714.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42039.log 2024-04-24T02:01:42.438 INFO:teuthology.orchestra.run.smithi079.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.43714.log.gz 2024-04-24T02:01:42.438 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.28223.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28452.log 2024-04-24T02:01:42.438 INFO:teuthology.orchestra.run.smithi079.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.28223.log.gz 2024-04-24T02:01:42.438 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.42039.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28957.log 2024-04-24T02:01:42.438 INFO:teuthology.orchestra.run.smithi079.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.42039.log.gz 2024-04-24T02:01:42.438 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.28452.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.26234.log 2024-04-24T02:01:42.438 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28452.log.gz 2024-04-24T02:01:42.438 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.28957.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30688.log 2024-04-24T02:01:42.438 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28957.log.gz 2024-04-24T02:01:42.438 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph.tmp-client.admin.26234.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29105.log 2024-04-24T02:01:42.439 INFO:teuthology.orchestra.run.smithi079.stderr: 66.7% -- replaced with /var/log/ceph/ceph.tmp-client.admin.26234.log.gz 2024-04-24T02:01:42.439 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.30688.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37250.log 2024-04-24T02:01:42.439 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30688.log.gz 2024-04-24T02:01:42.439 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.29105.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34567.log 2024-04-24T02:01:42.439 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29105.log.gz 2024-04-24T02:01:42.439 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.37250.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43763.log 2024-04-24T02:01:42.439 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.37250.log.gz 2024-04-24T02:01:42.439 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.34567.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43883.log 2024-04-24T02:01:42.439 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34567.log.gz 2024-04-24T02:01:42.439 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.43763.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36455.log 2024-04-24T02:01:42.439 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.43763.log.gz 2024-04-24T02:01:42.439 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.43883.log: gzip -5 --verbose 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43883.log.gz 2024-04-24T02:01:42.439 INFO:teuthology.orchestra.run.smithi079.stderr: -- /var/log/ceph/ceph-client.admin.32503.log 2024-04-24T02:01:42.439 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.36455.log: gzip 49.6% -- replaced with /var/log/ceph/ceph-client.admin.36455.log.gz 2024-04-24T02:01:42.439 INFO:teuthology.orchestra.run.smithi079.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.32463.log 2024-04-24T02:01:42.440 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.32503.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.32503.log.gz 2024-04-24T02:01:42.440 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30648.log 2024-04-24T02:01:42.440 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38605.log 2024-04-24T02:01:42.440 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.32463.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32463.log.gz 2024-04-24T02:01:42.440 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.30648.log: gzip -5 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30648.log.gz 2024-04-24T02:01:42.440 INFO:teuthology.orchestra.run.smithi079.stderr: --verbose -- /var/log/ceph/ceph-client.admin.31394.log 2024-04-24T02:01:42.440 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.38605.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29991.log 2024-04-24T02:01:42.440 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.38605.log.gz 2024-04-24T02:01:42.440 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.31394.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36981.log 2024-04-24T02:01:42.440 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31394.log.gz 2024-04-24T02:01:42.440 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.29991.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43843.log 2024-04-24T02:01:42.440 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29991.log.gz 2024-04-24T02:01:42.440 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.36981.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34845.log 2024-04-24T02:01:42.440 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.36981.log.gz 2024-04-24T02:01:42.441 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.43843.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37897.log 2024-04-24T02:01:42.441 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43843.log.gz 2024-04-24T02:01:42.441 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.34845.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43923.log 2024-04-24T02:01:42.441 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34845.log.gz 2024-04-24T02:01:42.441 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.37897.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29475.log 2024-04-24T02:01:42.441 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.37897.log.gz 2024-04-24T02:01:42.441 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.43923.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43505.log 2024-04-24T02:01:42.441 INFO:teuthology.orchestra.run.smithi079.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.43923.log.gz 2024-04-24T02:01:42.441 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.29475.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28786.log 2024-04-24T02:01:42.441 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29475.log.gz 2024-04-24T02:01:42.441 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.43505.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41010.log 2024-04-24T02:01:42.441 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43505.log.gz 2024-04-24T02:01:42.441 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.28786.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27971.log 2024-04-24T02:01:42.441 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28786.log.gz 2024-04-24T02:01:42.442 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.41010.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39081.log 2024-04-24T02:01:42.442 INFO:teuthology.orchestra.run.smithi079.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.41010.log.gz 2024-04-24T02:01:42.442 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.27971.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30600.log 2024-04-24T02:01:42.442 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.27971.log.gz 2024-04-24T02:01:42.442 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.39081.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41248.log 2024-04-24T02:01:42.442 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.39081.log.gz 2024-04-24T02:01:42.442 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.30600.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29189.log 2024-04-24T02:01:42.442 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30600.log.gz 2024-04-24T02:01:42.442 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.41248.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34707.log 2024-04-24T02:01:42.442 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41248.log.gz 2024-04-24T02:01:42.442 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.29189.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29648.log 2024-04-24T02:01:42.442 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29189.log.gz 2024-04-24T02:01:42.442 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.34707.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44907.log 2024-04-24T02:01:42.442 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34707.log.gz 2024-04-24T02:01:42.442 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.29648.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27929.log 2024-04-24T02:01:42.443 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29648.log.gz 2024-04-24T02:01:42.443 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.44907.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34219.log 2024-04-24T02:01:42.443 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44907.log.gz 2024-04-24T02:01:42.443 INFO:teuthology.orchestra.run.smithi079.stderr:gzip/var/log/ceph/ceph-client.admin.27929.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.45135.log 2024-04-24T02:01:42.443 INFO:teuthology.orchestra.run.smithi079.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.27929.log.gz 2024-04-24T02:01:42.443 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.34219.log: gzip -5 --verbose -- 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34219.log.gz 2024-04-24T02:01:42.443 INFO:teuthology.orchestra.run.smithi079.stderr: /var/log/ceph/ceph-client.admin.41954.log 2024-04-24T02:01:42.443 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.45135.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.45135.log.gz 2024-04-24T02:01:42.443 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38340.log 2024-04-24T02:01:42.443 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37299.log 2024-04-24T02:01:42.443 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.41954.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.41954.log.gz 2024-04-24T02:01:42.443 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.38340.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45580.log 2024-04-24T02:01:42.443 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.38340.log.gz 2024-04-24T02:01:42.443 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31701.log 2024-04-24T02:01:42.444 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.37299.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.37299.log.gz 2024-04-24T02:01:42.444 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.45580.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31173.log 2024-04-24T02:01:42.444 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.45580.log.gz 2024-04-24T02:01:42.444 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.31701.log: gzip -5 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31701.log.gz 2024-04-24T02:01:42.444 INFO:teuthology.orchestra.run.smithi079.stderr: --verbose -- /var/log/ceph/ceph-client.admin.38425.log 2024-04-24T02:01:42.444 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.31173.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41691.log 2024-04-24T02:01:42.444 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31173.log.gz 2024-04-24T02:01:42.444 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30220.log 2024-04-24T02:01:42.444 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.38425.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.38425.log.gz 2024-04-24T02:01:42.444 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.41691.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38952.log 2024-04-24T02:01:42.444 INFO:teuthology.orchestra.run.smithi079.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.41691.log.gz 2024-04-24T02:01:42.444 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.30220.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41598.log 2024-04-24T02:01:42.444 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30220.log.gz 2024-04-24T02:01:42.444 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.38952.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36306.log 2024-04-24T02:01:42.444 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.38952.log.gz 2024-04-24T02:01:42.445 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.41598.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44150.log 2024-04-24T02:01:42.445 INFO:teuthology.orchestra.run.smithi079.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.41598.log.gz 2024-04-24T02:01:42.445 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.36306.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32421.log 2024-04-24T02:01:42.445 INFO:teuthology.orchestra.run.smithi079.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.36306.log.gz 2024-04-24T02:01:42.445 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.44150.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28637.log 2024-04-24T02:01:42.445 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44150.log.gz 2024-04-24T02:01:42.445 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.32421.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31890.log 2024-04-24T02:01:42.445 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32421.log.gz 2024-04-24T02:01:42.445 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.28637.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38992.log 2024-04-24T02:01:42.445 INFO:teuthology.orchestra.run.smithi079.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28637.log.gz 2024-04-24T02:01:42.445 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.31890.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31890.log.gz 2024-04-24T02:01:42.445 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.38992.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.38992.log.gz 2024-04-24T02:01:42.586 INFO:teuthology.orchestra.run.smithi079.stderr: 89.9% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-04-24T02:01:43.877 INFO:teuthology.orchestra.run.smithi103.stderr: 92.6% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-04-24T02:01:43.961 INFO:teuthology.orchestra.run.smithi079.stderr: 92.6% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-04-24T02:01:44.062 INFO:teuthology.orchestra.run.smithi079.stderr: 92.5% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-04-24T02:01:44.064 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-24T02:01:44.064 INFO:teuthology.orchestra.run.smithi079.stderr:real 0m1.849s 2024-04-24T02:01:44.064 INFO:teuthology.orchestra.run.smithi079.stderr:user 0m3.810s 2024-04-24T02:01:44.064 INFO:teuthology.orchestra.run.smithi079.stderr:sys 0m0.308s 2024-04-24T02:01:44.149 INFO:teuthology.orchestra.run.smithi103.stderr: 92.4% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-04-24T02:01:44.151 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-04-24T02:01:44.152 INFO:teuthology.orchestra.run.smithi103.stderr:real 0m1.902s 2024-04-24T02:01:44.152 INFO:teuthology.orchestra.run.smithi103.stderr:user 0m3.441s 2024-04-24T02:01:44.152 INFO:teuthology.orchestra.run.smithi103.stderr:sys 0m0.141s 2024-04-24T02:01:44.152 INFO:tasks.ceph:Archiving logs... 2024-04-24T02:01:44.152 DEBUG:teuthology.misc:Transferring archived files from smithi079:/var/log/ceph to /home/teuthworker/archive/yingxin-2024-04-24_01:32:34-crimson-rados-ci-yingxin-crimson-load-obc-new-1-distro-default-smithi/7671138/remote/smithi079/log 2024-04-24T02:01:44.153 DEBUG:teuthology.orchestra.run.smithi079:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-24T02:01:45.151 DEBUG:teuthology.misc:Transferring archived files from smithi103:/var/log/ceph to /home/teuthworker/archive/yingxin-2024-04-24_01:32:34-crimson-rados-ci-yingxin-crimson-load-obc-new-1-distro-default-smithi/7671138/remote/smithi103/log 2024-04-24T02:01:45.152 DEBUG:teuthology.orchestra.run.smithi103:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-24T02:01:45.783 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-24T02:01:45.793 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_424d8c4389b684b0a4d72510e1b4d398d8e6c8f4/qa/tasks/rados.py", line 308, in task running.get() File "src/gevent/greenlet.py", line 831, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/gevent/_compat.py", line 49, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_ceph-c_424d8c4389b684b0a4d72510e1b4d398d8e6c8f4/qa/tasks/rados.py", line 289, in thread run.wait(tests.values()) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 179, in _raise_for_status raise CommandCrashedError(command=self.command) teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 0 --op copy_from 0 --op write_excl 50 --pool unique_pool_0' 2024-04-24T02:01:45.794 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-24T02:01:45.794 DEBUG:teuthology.orchestra.run.smithi079:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-24T02:01:45.797 DEBUG:teuthology.orchestra.run.smithi103:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-24T02:01:45.872 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-24T02:01:45.881 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-24T02:01:45.882 DEBUG:teuthology.orchestra.run.smithi079:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-24T02:01:45.884 DEBUG:teuthology.orchestra.run.smithi103:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-24T02:01:45.899 INFO:teuthology.orchestra.run.smithi103.stderr:bash: line 1: ntpq: command not found 2024-04-24T02:01:45.899 INFO:teuthology.orchestra.run.smithi079.stderr:bash: line 1: ntpq: command not found 2024-04-24T02:01:45.903 INFO:teuthology.orchestra.run.smithi103.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-24T02:01:45.903 INFO:teuthology.orchestra.run.smithi103.stdout:=============================================================================== 2024-04-24T02:01:45.903 INFO:teuthology.orchestra.run.smithi103.stdout:^- hv01.front.sepia.ceph.com 2 6 377 31 +1426us[+1434us] +/- 583ms 2024-04-24T02:01:45.903 INFO:teuthology.orchestra.run.smithi103.stdout:^* hv02.front.sepia.ceph.com 3 6 377 29 -578us[ -570us] +/- 61ms 2024-04-24T02:01:45.903 INFO:teuthology.orchestra.run.smithi103.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 32 +776us[ +784us] +/- 73ms 2024-04-24T02:01:45.904 INFO:teuthology.orchestra.run.smithi103.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T02:01:45.904 INFO:teuthology.orchestra.run.smithi079.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-24T02:01:45.904 INFO:teuthology.orchestra.run.smithi079.stdout:=============================================================================== 2024-04-24T02:01:45.905 INFO:teuthology.orchestra.run.smithi079.stdout:^- hv01.front.sepia.ceph.com 2 6 377 30 +1398us[+1405us] +/- 583ms 2024-04-24T02:01:45.905 INFO:teuthology.orchestra.run.smithi079.stdout:^* hv02.front.sepia.ceph.com 3 6 377 30 -598us[ -591us] +/- 61ms 2024-04-24T02:01:45.905 INFO:teuthology.orchestra.run.smithi079.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 28 +745us[ +745us] +/- 73ms 2024-04-24T02:01:45.905 INFO:teuthology.orchestra.run.smithi079.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T02:01:45.905 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-24T02:01:45.915 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-24T02:01:45.916 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-24T02:01:45.925 DEBUG:teuthology.orchestra.run.smithi079:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-24T02:01:46.016 DEBUG:teuthology.orchestra.run.smithi103:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-24T02:01:46.075 DEBUG:teuthology.orchestra.run.smithi079:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-24T02:01:46.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T02:01:46.103 DEBUG:teuthology.orchestra.run.smithi103:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-24T02:01:46.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T02:01:46.129 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-24T02:01:46.139 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-24T02:01:46.147 INFO:teuthology.task.internal:Duration was 827.001239 seconds 2024-04-24T02:01:46.147 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-24T02:01:46.155 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-24T02:01:46.155 DEBUG:teuthology.orchestra.run.smithi079:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-24T02:01:46.158 DEBUG:teuthology.orchestra.run.smithi103:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-24T02:01:46.193 INFO:teuthology.orchestra.run.smithi079.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-24T02:01:46.205 INFO:teuthology.orchestra.run.smithi103.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-24T02:01:46.389 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-24T02:01:46.389 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi079.front.sepia.ceph.com 2024-04-24T02:01:46.389 DEBUG:teuthology.orchestra.run.smithi079:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-24T02:01:46.449 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi103.front.sepia.ceph.com 2024-04-24T02:01:46.449 DEBUG:teuthology.orchestra.run.smithi103:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-24T02:01:46.476 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-24T02:01:46.476 DEBUG:teuthology.orchestra.run.smithi079:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-24T02:01:46.491 DEBUG:teuthology.orchestra.run.smithi103:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-24T02:01:46.540 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-24T02:01:46.540 DEBUG:teuthology.orchestra.run.smithi079:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-24T02:01:46.556 DEBUG:teuthology.orchestra.run.smithi103:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-24T02:01:46.673 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-24T02:01:46.682 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-24T02:01:46.683 DEBUG:teuthology.orchestra.run.smithi079:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-24T02:01:46.706 DEBUG:teuthology.orchestra.run.smithi103:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-24T02:01:46.730 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-24T02:01:46.739 DEBUG:teuthology.orchestra.run.smithi079:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-24T02:01:46.749 DEBUG:teuthology.orchestra.run.smithi103:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-24T02:01:46.768 INFO:teuthology.orchestra.run.smithi079.stdout:kernel.core_pattern = core 2024-04-24T02:01:46.792 INFO:teuthology.orchestra.run.smithi103.stdout:kernel.core_pattern = core 2024-04-24T02:01:46.822 DEBUG:teuthology.orchestra.run.smithi079:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-24T02:01:46.841 WARNING:teuthology.task.internal:Found coredumps on ubuntu@smithi079.front.sepia.ceph.com, flagging run as failed 2024-04-24T02:01:46.842 DEBUG:teuthology.orchestra.run.smithi103:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-24T02:01:46.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T02:01:46.876 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-24T02:01:46.887 INFO:teuthology.task.internal:Transferring archived files... 2024-04-24T02:01:46.888 DEBUG:teuthology.misc:Transferring archived files from smithi079:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yingxin-2024-04-24_01:32:34-crimson-rados-ci-yingxin-crimson-load-obc-new-1-distro-default-smithi/7671138/remote/smithi079 2024-04-24T02:01:46.888 DEBUG:teuthology.orchestra.run.smithi079:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-24T02:02:00.434 INFO:teuthology.task.internal:Transferring binaries for coredumps... 2024-04-24T02:02:00.446 INFO:teuthology.task.internal: core looks like: /home/teuthworker/archive/yingxin-2024-04-24_01:32:34-crimson-rados-ci-yingxin-crimson-load-obc-new-1-distro-default-smithi/7671138/remote/smithi079/coredump/1713924059.29762.core.gz: gzip compressed data, was "1713924059.29762.core", last modified: Wed Apr 24 02:01:46 2024, max compression, original size modulo 2^32 346198016 2024-04-24T02:02:00.446 INFO:teuthology.task.internal:core is compressed, try accessing gzip file ... 2024-04-24T02:02:01.829 INFO:teuthology.task.internal: core looks like: /tmp/tmp8ab0efcc: ELF 64-bit LSB core file, x86-64, version 1 (SYSV), SVR4-style, from 'ceph_test_rados --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 -', real uid: 1000, effective uid: 1000, real gid: 1316, effective gid: 1316, execfn: '/usr/bin/ceph_test_rados', platform: 'x86_64' 2024-04-24T02:02:01.863 INFO:teuthology.task.internal: dump_program: ceph_test_rados --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 - 2024-04-24T02:02:01.864 DEBUG:teuthology.orchestra.run.smithi079:> which 'ceph_test_rados --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 -' 2024-04-24T02:02:01.945 INFO:teuthology.orchestra.run.smithi079.stderr:/usr/bin/which: unrecognized option '--max-ops' 2024-04-24T02:02:01.945 INFO:teuthology.orchestra.run.smithi079.stderr:/usr/bin/which: unrecognized option '--objects' 2024-04-24T02:02:01.945 INFO:teuthology.orchestra.run.smithi079.stderr:/usr/bin/which: unrecognized option '--max-in-flight' 2024-04-24T02:02:01.945 INFO:teuthology.orchestra.run.smithi079.stderr:/usr/bin/which: unrecognized option '--size' 2024-04-24T02:02:01.945 INFO:teuthology.orchestra.run.smithi079.stderr:/usr/bin/which: no 4000 in (/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin) 2024-04-24T02:02:01.945 INFO:teuthology.orchestra.run.smithi079.stderr:/usr/bin/which: no 50 in (/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin) 2024-04-24T02:02:01.945 INFO:teuthology.orchestra.run.smithi079.stderr:/usr/bin/which: no 16 in (/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin) 2024-04-24T02:02:01.945 INFO:teuthology.orchestra.run.smithi079.stderr:/usr/bin/which: no 4000000 in (/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin) 2024-04-24T02:02:01.945 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph_test_rados 2024-04-24T02:02:01.946 DEBUG:teuthology.orchestra.run:got remote process result: 5 2024-04-24T02:02:01.946 INFO:teuthology.orchestra.run.smithi079.stderr:/usr/bin/which: no - in (/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin) 2024-04-24T02:02:01.947 ERROR:teuthology.run_tasks:Manager failed: internal.archive Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/internal/__init__.py", line 392, in archive yield File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_424d8c4389b684b0a4d72510e1b4d398d8e6c8f4/qa/tasks/rados.py", line 308, in task running.get() File "src/gevent/greenlet.py", line 831, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/gevent/_compat.py", line 49, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_ceph-c_424d8c4389b684b0a4d72510e1b4d398d8e6c8f4/qa/tasks/rados.py", line 289, in thread run.wait(tests.values()) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 179, in _raise_for_status raise CommandCrashedError(command=self.command) teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 0 --op copy_from 0 --op write_excl 50 --pool unique_pool_0' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/internal/__init__.py", line 420, in archive fetch_binaries_for_coredumps(path, rem) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/internal/__init__.py", line 345, in fetch_binaries_for_coredumps remote_path = remote.sh(['which', dump_program]).rstrip() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi079 with status 5: "which 'ceph_test_rados --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 -'" 2024-04-24T02:02:01.949 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-24T02:02:01.972 INFO:teuthology.task.internal:Not uploading archives. 2024-04-24T02:02:01.973 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-24T02:02:01.985 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-24T02:02:01.985 DEBUG:teuthology.orchestra.run.smithi079:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-24T02:02:01.988 DEBUG:teuthology.orchestra.run.smithi103:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-24T02:02:02.003 INFO:teuthology.orchestra.run.smithi079.stdout: 262213 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Apr 24 02:01 /home/ubuntu/cephtest 2024-04-24T02:02:02.003 INFO:teuthology.orchestra.run.smithi079.stdout: 262214 4 drwxr-xr-x 5 ubuntu ubuntu 4096 Apr 24 02:01 /home/ubuntu/cephtest/archive 2024-04-24T02:02:02.003 INFO:teuthology.orchestra.run.smithi079.stdout: 524289 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 24 02:01 /home/ubuntu/cephtest/archive/audit 2024-04-24T02:02:02.003 INFO:teuthology.orchestra.run.smithi079.stdout: 524291 144 -rw------- 1 ubuntu ubuntu 145856 Apr 24 02:01 /home/ubuntu/cephtest/archive/audit/audit.log.gz 2024-04-24T02:02:02.003 INFO:teuthology.orchestra.run.smithi079.stdout: 262216 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 24 02:01 /home/ubuntu/cephtest/archive/syslog 2024-04-24T02:02:02.003 INFO:teuthology.orchestra.run.smithi079.stdout: 262259 4 -rw-rw-rw- 1 ubuntu ubuntu 29 Apr 24 01:47 /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2024-04-24T02:02:02.003 INFO:teuthology.orchestra.run.smithi079.stdout: 262217 188 -rw-r--r-- 1 ubuntu ubuntu 188519 Apr 24 02:01 /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-24T02:02:02.003 INFO:teuthology.orchestra.run.smithi079.stdout: 262218 4 -rw-rw-rw- 1 ubuntu ubuntu 29 Apr 24 01:47 /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2024-04-24T02:02:02.003 INFO:teuthology.orchestra.run.smithi079.stdout: 262215 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 24 02:00 /home/ubuntu/cephtest/archive/coredump 2024-04-24T02:02:02.003 INFO:teuthology.orchestra.run.smithi079.stdout: 266825 232736 -rw------- 1 ubuntu ubuntu 346198016 Apr 24 02:00 /home/ubuntu/cephtest/archive/coredump/1713924059.29762.core 2024-04-24T02:02:02.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T02:02:02.004 INFO:teuthology.orchestra.run.smithi079.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-04-24T02:02:02.005 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/internal/__init__.py", line 48, in base yield File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_424d8c4389b684b0a4d72510e1b4d398d8e6c8f4/qa/tasks/rados.py", line 308, in task running.get() File "src/gevent/greenlet.py", line 831, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/gevent/_compat.py", line 49, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_ceph-c_424d8c4389b684b0a4d72510e1b4d398d8e6c8f4/qa/tasks/rados.py", line 289, in thread run.wait(tests.values()) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 179, in _raise_for_status raise CommandCrashedError(command=self.command) teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 0 --op copy_from 0 --op write_excl 50 --pool unique_pool_0' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi079 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-04-24T02:02:02.005 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-24T02:02:02.015 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-24T02:02:02.026 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 0 --op copy_from 0 --op write_excl 50 --pool unique_pool_0' 2024-04-24T02:02:02.027 INFO:teuthology.nuke:Checking targets against current locks 2024-04-24T02:02:02.031 INFO:teuthology.orchestra.run.smithi103.stdout: 262213 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Apr 24 02:01 /home/ubuntu/cephtest 2024-04-24T02:02:02.031 INFO:teuthology.orchestra.run.smithi103.stdout: 262214 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Apr 24 02:01 /home/ubuntu/cephtest/archive 2024-04-24T02:02:02.031 INFO:teuthology.orchestra.run.smithi103.stdout: 401286 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 24 02:01 /home/ubuntu/cephtest/archive/audit 2024-04-24T02:02:02.031 INFO:teuthology.orchestra.run.smithi103.stdout: 401569 108 -rw------- 1 ubuntu ubuntu 107531 Apr 24 02:01 /home/ubuntu/cephtest/archive/audit/audit.log.gz 2024-04-24T02:02:02.031 INFO:teuthology.orchestra.run.smithi103.stdout: 262216 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 24 02:01 /home/ubuntu/cephtest/archive/syslog 2024-04-24T02:02:02.031 INFO:teuthology.orchestra.run.smithi103.stdout: 262259 4 -rw-rw-rw- 1 ubuntu ubuntu 29 Apr 24 01:47 /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2024-04-24T02:02:02.031 INFO:teuthology.orchestra.run.smithi103.stdout: 262217 176 -rw-r--r-- 1 ubuntu ubuntu 179429 Apr 24 02:01 /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-24T02:02:02.031 INFO:teuthology.orchestra.run.smithi103.stdout: 262218 4 -rw-rw-rw- 1 ubuntu ubuntu 29 Apr 24 01:47 /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2024-04-24T02:02:02.032 INFO:teuthology.orchestra.run.smithi103.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-04-24T02:02:02.053 DEBUG:teuthology.nuke:shortname: smithi079 2024-04-24T02:02:02.053 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-24T02:02:02.077 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi079.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yingxin-2024-04-24_01:32:34-crimson-rados-ci-yingxin-crimson-load-obc-new-1-distro-default-smithi/7671138', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-24 01:42:05.650772', 'locked_by': 'scheduled_yingxin@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBExTLpkVK6cnU8uvK6oiGFbBNE58CQpG+zqhnOeuD48hfXiTHMTNkdoxAlH5naPPwvvTytBB/PBvw2/kSuQQPGA='} 2024-04-24T02:02:02.080 DEBUG:teuthology.nuke:shortname: smithi103 2024-04-24T02:02:02.080 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-24T02:02:02.101 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi103.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yingxin-2024-04-24_01:32:34-crimson-rados-ci-yingxin-crimson-load-obc-new-1-distro-default-smithi/7671138', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-24 01:42:05.652403', 'locked_by': 'scheduled_yingxin@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJwP7wGqBu8yx2tmUKdALm5FUtcVc092/3lmeqecFmPUCzjl9i77SkaJVwmYpo5zeySSoUJukG4ObFttE6oivLk='} 2024-04-24T02:02:02.128 INFO:teuthology.orchestra.console.smithi079:Power off 2024-04-24T02:02:02.129 DEBUG:teuthology.orchestra.console.smithi079:pexpect command: ipmitool -H smithi079.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-24T02:02:02.155 DEBUG:teuthology.orchestra.console.smithi079:power off output: Chassis Power Control: Down/Off 2024-04-24T02:02:02.155 DEBUG:teuthology.orchestra.console.smithi079:pexpect command: ipmitool -H smithi079.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-24T02:02:02.169 INFO:teuthology.orchestra.console.smithi103:Power off 2024-04-24T02:02:02.170 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-24T02:02:02.179 DEBUG:teuthology.orchestra.console.smithi079:check power output: Chassis Power is on 2024-04-24T02:02:02.194 DEBUG:teuthology.orchestra.console.smithi103:power off output: Chassis Power Control: Down/Off 2024-04-24T02:02:02.194 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-24T02:02:02.217 DEBUG:teuthology.orchestra.console.smithi103:check power output: Chassis Power is on 2024-04-24T02:02:06.179 DEBUG:teuthology.orchestra.console.smithi079:pexpect command: ipmitool -H smithi079.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-24T02:02:06.218 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-24T02:02:06.290 DEBUG:teuthology.orchestra.console.smithi079:check power output: Chassis Power is on 2024-04-24T02:02:06.327 DEBUG:teuthology.orchestra.console.smithi103:check power output: Chassis Power is on 2024-04-24T02:02:10.292 DEBUG:teuthology.orchestra.console.smithi079:pexpect command: ipmitool -H smithi079.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-24T02:02:10.328 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-24T02:02:10.402 DEBUG:teuthology.orchestra.console.smithi079:check power output: Chassis Power is on 2024-04-24T02:02:10.437 DEBUG:teuthology.orchestra.console.smithi103:check power output: Chassis Power is off 2024-04-24T02:02:10.537 INFO:teuthology.orchestra.console.smithi103:Power off completed 2024-04-24T02:02:14.406 DEBUG:teuthology.orchestra.console.smithi079:pexpect command: ipmitool -H smithi079.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-24T02:02:14.516 DEBUG:teuthology.orchestra.console.smithi079:check power output: Chassis Power is off 2024-04-24T02:02:14.617 INFO:teuthology.orchestra.console.smithi079:Power off completed 2024-04-24T02:02:14.722 INFO:teuthology.run:Summary data: description: crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/simple thrashosds-health workloads/snaps-few-objects} duration: 827.001238822937 failure_reason: 'Command crashed: ''CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 0 --op copy_from 0 --op write_excl 50 --pool unique_pool_0''' flavor: crimson owner: scheduled_yingxin@teuthology status: fail success: false 2024-04-24T02:02:14.722 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-24T02:02:14.797 INFO:teuthology.run:FAIL